OpenContracts

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

OpenContracts is an MIT-licensed enterprise document analytics platform for PDFs and text-based formats. It features a Django/GraphQL backend with PostgreSQL + pgvector, a React/TypeScript frontend with Jotai state management, and pluggable document processing pipelines powered by machine learning models.

## Baseline Commit Rules
1. Always ensure all affected (or new) tests pass - backend tests suite should only be run in its entirety for good reason as it takes 30+ minutes.
2. Always make sure typescript compiles and pre-commits pass before committing new code.
3. Never credit Claude or Claude Code in commit messages, PR messages, comments, or any other artifacts. This includes Co-Authored-By lines, "Generated by Claude", etc.

## Essential Commands

### Backend (Django)

```bash
# Run backend tests (sequential, use --keepdb to speed up subsequent runs)
docker compose -f test.yml run django python manage.py test --keepdb

# Run backend tests in PARALLEL (recommended - ~4x faster)
# Uses pytest-xdist with 4 workers, --dist loadscope keeps class tests together
docker compose -f test.yml run django pytest -n 4 --dist loadscope

# Run parallel tests with auto-detected worker count (uses all CPU cores)
docker compose -f test.yml run django pytest -n auto --dist loadscope

# Run parallel tests with fresh databases (first run or after schema changes)
docker compose -f test.yml run django pytest -n 4 --dist loadscope --create-db

# Run specific test file
docker compose -f test.yml run django python manage.py test opencontractserver.tests.test_notifications --keepdb

# Run specific test file in parallel
docker compose -f test.yml run django pytest opencontractserver/tests/test_notifications.py -n 4 --dist loadscope

# Run specific test class/method
docker compose -f test.yml run django python manage.py test opencontractserver.tests.test_notifications.TestNotificationModel.test_create_notification --keepdb

# Apply database migrations
docker compose -f local.yml run django python manage.py migrate

# Create new migration
docker compose -f local.yml run django python manage.py makemigrations

# Django shell
docker compose -f local.yml run django python manage.py shell

# Code quality (runs automatically via pre-commit hooks)
pre-commit run --all-files
```

### Frontend (React/TypeScript)

```bash
cd frontend

# Start development server (proxies to Django on :8000)
yarn start

# Run unit tests (Vitest) - watches by default
yarn test:unit

# Run component tests (Playwright) - CRITICAL: Use --reporter=list to prevent hanging
yarn test:ct --reporter=list

# Run component tests with grep filter
yarn test:ct --reporter=list -g "test name pattern"

# Run E2E tests
yarn test:e2e

# Coverage reports (unit tests via Vitest, component tests via Playwright + Istanbul)
yarn test:coverage:unit
yarn test:coverage:ct

# Linting and formatting
yarn lint
yarn fix-styles

# Build for production
yarn build

# Preview production build locally
yarn serve
```

### Production Deployment

```bash
# CRITICAL: Always run migrations FIRST in production
docker compose -f production.yml --profile migrate up migrate

# Then start main services
docker compose -f production.yml up
```

## High-Level Architecture

### Backend Architecture

**Stack**: Django 4.x + GraphQL (strawberry-graphql) + PostgreSQL + pgvector + Celery

**Key Patterns**:

1. **GraphQL Schema Organization** (strawberry; migrated from graphene — query shapes pinned):
   - `config/graphql/core/` - shared runtime: relay global IDs/`Node`, connection factories + graphene-parity pagination, FilterSet arg mapping, scalars, permission resolvers, DRF mutation bases, auth decorators
   - `config/graphql/*_types.py` - GraphQL type definitions (organized by feature)
   - `config/graphql/*_queries.py` / `*_mutations.py` - Query/Mutation fields + resolvers (each module exports `QUERY_FIELDS` / `MUTATION_FIELDS`)
   - `config/graphql/schema.py` - Schema composition + validation rules
   - `config/graphql/schema.graphql` - **golden SDL contract**; `opencontractserver/tests/test_schema_parity.py` fails on ANY shape drift. Regenerate it deliberately when changing the API surface (command in that test's docstring).
   - `config/graphql/views.py` - HTTP view; per-request auth (JWT/Auth0/API-key) happens in `get_context` via `AUTHENTICATION_BACKENDS` (no per-resolver auth middleware)

2. **Permission System** (CRITICAL - see `docs/permissioning/consolidated_permissioning_guide.md`):
   - **Annotations & Relationships**: NO individual permissions - inherited from document + corpus
   - **Documents & Corpuses**: Direct object-level permissions via django-guardian
   - **Analyses & Extracts**: Hybrid model (own permissions + corpus permissions + document filtering)
   - Formula: `Effective Permission = MIN(document_permission, corpus_permission)`
   - **Structural items are ALWAYS read-only** except for superusers
   - Use `Model.objects.visible_to_user(user)` pattern (NOT `resolve_oc_model_queryset` - DEPRECATED)
   - For corpus-scoped document access (the most common pattern), prefer `CorpusDocumentService.get_corpus_documents(user, corpus)` over composing `visible_to_user` filters by hand — the service is the canonical entry point and prevents IDOR-prone copy-paste fusions of `corpus.get_documents()` + `Document.objects.visible_to_user(user)`.
   - **Corpus document access — two deliberate semantics (issue #1682)**: `CorpusDocumentService.get_corpus_documents(user, corpus)` is **corpus-as-gate** — corpus READ unlocks *every* document with an active path in that corpus. It is the documented default for pipeline-facing callers (MCP, badge/analysis tasks) that legitimately operate over a whole readable corpus and never return or persist verbatim document content to the caller. `CorpusDocumentService.get_corpus_documents_visible_to_user(user, corpus)` enforces **`MIN(document_permission, corpus_permission)`** — a private document inside a public (or merely shared) corpus stays hidden from users who lack document-level READ. User-facing surfaces that must not leak private documents (e.g. the GraphQL `CorpusType.documents` resolver) MUST use the `_visible_to_user` variant. **Authority enrichment is `_visible_to_user`, not corpus-as-gate** (PR #2084): `EnrichmentService.discover`/`scan`/`apply` (`opencontractserver/enrichment/services/enrichment_service.py::_load`) return verbatim excerpts and persist `Annotation`/`CorpusReference` rows derived from document text, so they load through the MIN variant and surface a `documents_excluded_by_visibility` count (+ a WARNING) when the caller's `creator_id` lacks per-document READ on some documents. Choose the method by caller intent; never silently swap one semantic for the other.

3. **Permission-annotation fields** (`myPermissions`, `isPublished`, `objectSharedWith`):
   - Resolved by `config/graphql/core/permissions.py` (port of the graphene-era `AnnotatePermissionsForReadMixin`); most GraphQL types expose them
   - Requires model to have guardian permission tables (`{model}userobjectpermission_set`)
   - Per-request model-permission maps are memoised on `info.context.permission_annotations`
   - Notifications use simple ownership model and DON'T expose these fields

4. **Django Signal Handlers**:
   - Automatic notification creation on model changes (see `opencontractserver/notifications/signals.py`)
   - Must be imported in app's `apps.py` `ready()` method
   - Use `_skip_signals` attribute to prevent duplicate notifications in tests

5. **Pluggable Parser Pipeline**:
   - Base classes in `opencontractserver/pipeline/base/`
   - Parsers, embedders, thumbnailers auto-discovered and registered
   - Multiple backends: Docling (ML-based), LlamaParse, Text
   - All convert to unified PAWLs format for frontend

6. **Agent Tool Architecture** (see `docs/architecture/llms/README.md`):
   - `CoreTool` (framework-agnostic) → `PydanticAIToolWrapper` (pydantic-ai specific)
   - **All production tools MUST be async** (`a`-prefixed in `core_tools.py`). The wrapper supports sync functions for lightweight helpers/tests but does NOT wrap them in a thread pool — sync ORM calls will raise `SynchronousOnlyOperation`.
   - Tool fault tolerance (issue #820): operational exceptions are caught and returned as error strings to the LLM; security exceptions (`PermissionError`, `ToolConfirmationRequired`) propagate.
   - Pre-execution checks run on every call (not cached): permission validation, resource ID validation, approval gates.

### Frontend Architecture

**Stack**: React 18 + TypeScript + Apollo Client + Jotai (atoms) + PDF.js + Vite

**Key Patterns**:

1. **State Management - Jotai Atoms**:
   - **Global state via atoms** in `frontend/src/atoms/` (NOT Redux/Context)
   - Key atoms: `selectedFolderIdAtom`, `folderCorpusIdAtom`, `selectedMessageIdAtom`
   - Derived atoms automatically update when dependencies change
   - Apollo reactive vars in `frontend/src/graphql/cache.ts` for UI state
   - AuthGate pattern ensures auth completes before rendering

2. **Central Routing System** (see `docs/frontend/routing_system.md`):
   - Single source of truth: `frontend/src/routing/CentralRouteManager.tsx`
   - URL paths → Entity resolution via GraphQL slug queries
   - URL params ↔ Reactive vars (bidirectional sync)
   - Components consume state via reactive vars, never touch URLs directly
   - Deep linking and canonical redirects handled automatically

3. **PDF Annotation System** (see `.cursor/rules/pdf-viewer-and-annotator-architecture.mdc`):
   - **Virtualized rendering**: Only visible pages (+overscan) rendered for performance
   - Binary search to find visible page range (O(log n))
   - Height caching per zoom level
   - Two-phase scroll-to-annotation system
   - Dual-layer architecture: Document layer (annotations) + Knowledge layer (summaries)

4. **Unified Filtering Architecture**:
   - `useVisibleAnnotations` hook provides single-source-of-truth filtering for both annotations and relationship-connected annotations
   - Reads from Jotai atoms via `useAnnotationDisplay`, `useAnnotationControls`, and `useAnnotationSelection` (from `UISettingsAtom`)
   - Ensures consistency across all components
   - Forced visibility for selected items and their relationship connections

5. **Component Testing** (see `.cursor/rules/test-document-knowledge-base.mdc`):
   - **ALWAYS mount components through test wrappers** (e.g., `DocumentKnowledgeBaseTestWrapper`)
   - Wrapper provides: MockedProvider + InMemoryCache + Jotai Provider + asset mocking
   - **Use `--reporter=list` flag to prevent hanging**
   - Increase timeouts (20s+) for PDF rendering in Chromium
   - GraphQL mocks must match variables EXACTLY (null vs undefined matters)
   - Mock same query multiple times for refetches
   - Use `page.mouse` for PDF canvas interactions (NOT `locator.dragTo`)
   - Add settle time after drag operations (500ms UI, 1000ms Apollo cache)

6. **Development Server Configuration**:
   - Vite dev server on :3000 proxies to Django on :8000
   - WebSocket proxy for `/ws` → `ws://localhost:8000`
   - GraphQL proxy for `/graphql` → `http://localhost:8000`
   - REST API proxy for `/api` → `http://localhost:8000`
   - Auth0 optional via `REACT_APP_USE_AUTH0` environment variable

### Data Flow Architecture

**Document Processing**:
1. Upload → Parser Selection (Docling/LlamaParse/Text)
2. Parser generates PAWLs JSON (tokens with bounding boxes)
3. Text layer extracted from PAWLs
4. Annotations created for structure (headers, sections, etc.)
5. Relationships detected between elements
6. Vector embeddings generated for search

**GraphQL Permission Flow**:
1. Query resolver filters objects with `.visible_to_user(user)`
2. GraphQL types resolve `my_permissions` via `AnnotatePermissionsForReadMixin`
3. Frontend uses permissions to enable/disable UI features
4. Mutations check permissions and return consistent errors to prevent IDOR

## Critical Security Patterns

1. **IDOR Prevention**:
   - Query by both ID AND user-owned field: `Model.objects.get(pk=pk, recipient=user)`
   - Return same error message whether object doesn't exist or belongs to another user
   - Prevents enumeration via timing or different error messages

2. **Permission Checks**:
   - NEVER trust frontend - always check server-side
   - Use `Model.objects.visible_to_user(user)` manager method for querysets (list/filter)
   - For a single-object check use `Model.objects.user_can(user, obj, perm)` / `obj.user_can(user, perm)` — the canonical API, paired with `visible_to_user` and pinned to agree by `opencontractserver/tests/permissioning/test_authorization_invariants.py`. In GraphQL resolvers/mutations pass `request=info.context` to engage the Tier-2 permission cache; omit it in Celery/agent/internal code.

3. **XSS Prevention**:
   - User-generated content in JSON fields must be escaped on frontend
   - GraphQL's GenericScalar handles JSON serialization safely
   - Document this requirement in resolver comments

## Critical Concepts
1. No dead code - when deprecating or replacing code, always try to fully replace older code and, once it's no longer in use, delete it and related texts.
2. DRY - please always architect code for maximal dryness and always see if you can consolidate related code and remove duplicative code.
3. Single Responsibility Principle - Generally, ensure that each module / script has a single purpose or related purpose.
4. No magic numbers - we have constants files in opencontractserver/constants/ (backend) and frontend/src/assets/configurations/constants.ts (frontend). Use them for any hardcoded values.
5. Don't touch old tests without permission - if pre-existing tests fail after changes, try to identify why and present user with root cause analysis. If the test logic is correct but expectations need updating due to intentional behavior changes, document the change clearly.
6. Utility functions belong in utility files:
   - **Before writing new utilities**: Check existing utility files first (`frontend/src/utils/`, `opencontractserver/utils/`)
   - **When reviewing code**: If you find utility functions defined inline in components/views, check if they already exist in utility files. If not, consider whether they should be extracted there for reuse.
   - **Frontend utilities**: `frontend/src/utils/formatters.ts` (formatting), `frontend/src/utils/files.ts` (file operations), etc.
   - **Backend utilities**: `opencontractserver/utils/` contains permissioning, PDF processing, and other shared utilities
7. **Always go through the app's `services/` package.** Code with a user context (GraphQL resolvers, MCP tools, LLM tools, REST views, Celery tasks invoked with a user) **must** reach models through `opencontractserver/<app>/services/` — never compose `visible_to_user` / `user_can` / `user_has_permission_for_obj` inline. The shared base (`opencontractserver.shared.services.base.BaseService`) exposes `get_or_none`, `filter_visible`, `require_permission`, and `user_has` for the cases where a dedicated per-app method is overkill. The invariant is enforced twice — by a pytest test (`opencontractserver/tests/architecture/test_graphql_service_layer.py`) and by a Django system check (`opencontractserver/shared/checks.py`, `opencontracts.E001`) that fails `manage.py` startup on any inline Tier-0 use. **Scope of mechanical enforcement:** both the test and the check scan **`config/graphql/` only** (recursively). The rule above is *policy* for the other user-context surfaces — MCP tools (`opencontractserver/mcp/`), LLM tools (`opencontractserver/llms/tools/`), REST views, and user-context Celery tasks — but those are **not** scanned today and still contain correct-but-inline Tier-0 calls. Treat a green E001 as "no inline Tier-0 in `config/graphql/`," not "none anywhere." Failure messages carry the copy-pasteable recipe inline; see `docs/development/architecture_invariants.md` for the invariant catalogue and `docs/architecture/query_permission_patterns.md` for the per-app service catalogue + migration recipes.
8. **Docs: keep them current, concise, and pointer-based.** When you change behavior, update the relevant doc (or add one if none fits) as part of the same change — stale docs are worse than none. But keep docs *concise and prune as you go*: a doc that only ever grows rots into noise, so trim superseded content rather than appending forever. **Favor code pointers over pasted code** — reference files + symbols (e.g. `opencontractserver/enrichment/authorities.py::bootstrap_authority_corpus`) instead of copying snippets in. Pasted code drifts out of sync the moment the source changes and becomes dead documentation; a pointer stays live. (See `## Documentation Locations` for where things live, and `## Changelog Maintenance` for change records.)

## Testing Patterns

### Manual Test Scripts

**Location**: `docs/test_scripts/`

When performing manual testing (e.g., testing migrations, verifying database state, testing API endpoints interactively), **always document the test steps** in a markdown file under `docs/test_scripts/`. These scripts will later be used to build automated integration tests.

**Format**:
```markdown
# Test: [Brief description]

## Purpose
What this test verifies.

## Prerequisites
- Required state (e.g., "migration at 0058")
- Required data (e.g., "at least one document exists")

## Steps
1. Step one with exact command
   ```bash
   docker compose -f local.yml run --rm django python manage.py shell -c "..."
   ```
2. Step two...

## Expected Results
- What should happen after each step
- Success criteria

## Cleanup
Commands to restore original state if needed.
```

**When to document**:
- Migration testing (rollback, create test data, migrate forward)
- Database constraint validation
- Race condition verification
- Any manual verification requested during code review

### Backend Tests

**Location**: `opencontractserver/tests/`

**Parallel Testing** (pytest-xdist):
- Run with `-n 4` (or `-n auto`) for parallel execution across workers
- Use `--dist loadscope` to keep tests from the same class on the same worker (respects `setUpClass`)
- Each worker gets its own database (test_db_gw0, test_db_gw1, etc.)
- Use `@pytest.mark.serial` to mark tests that cannot run in parallel
- First run or after DB schema changes: add `--create-db` flag

**Patterns**:
- Use `TransactionTestCase` for tests with signals/asynchronous behavior
- Use `TestCase` for faster tests without transaction isolation
- Clear auto-created notifications when testing moderation: `Notification.objects.filter(recipient=user).delete()`
- Use `_skip_signals` attribute on instances to prevent signal handlers during fixtures

### Frontend Component Tests

**Location**: `frontend/tests/`

**Critical Requirements**:
- Mount through test wrappers that provide all required context
- GraphQL mocks must match query variables exactly
- Include mocks for empty-string variants (unexpected boot calls)
- Wait for visible evidence, not just network-idle
- Use `page.mouse` for PDF canvas interactions (NOT `locator.dragTo`)
- Add settle time after drag operations (500ms UI, 1000ms Apollo cache)

**Test Wrapper Pattern**:
```typescript
// ALWAYS use wrappers, never mount components directly
const component = await mount(
  <DocumentKnowledgeBaseTestWrapper corpusId="corpus-1" documentId="doc-1">
    <DocumentKnowledgeBase />
  </DocumentKnowledgeBaseTestWrapper>
);
```

### Automated Documentation Screenshots

**Location**: `docs/assets/images/screenshots/auto/` (output) | `frontend/tests/utils/docScreenshot.ts` (utility)

Screenshots for documentation are captured during Playwright component tests and pushed back to a PR branch by the `screenshots.yml` workflow. The workflow is **manual / on-demand only** (`workflow_dispatch`) — it does not auto-trigger on PR push or merge. Run it from the Actions tab or via `gh workflow run "Update Documentation Screenshots" -f pr_number=<N>` when you want to refresh the screenshots on a PR.

**How it works**:
1. Import `docScreenshot` from `./utils/docScreenshot` in any `.ct.tsx` test file
2. Call `await docScreenshot(page, "area--component--state")` after the component reaches the desired visual state
3. Reference the image in markdown: `![Alt text](../assets/images/screenshots/auto/area--component--state.png)`
4. Trigger the screenshot workflow on demand for the PR; it captures screenshots and commits any changes back to the PR branch

**Naming convention** (`--` separates segments, `-` within words):

| Segment | Purpose | Examples |
|---------|---------|----------|
| `area` | Feature area | `landing`, `badges`, `corpus`, `versioning`, `annotations` |
| `component` | Specific view or component | `hero-section`, `celebration-modal`, `list-view` |
| `state` | Visual state captured | `anonymous`, `with-data`, `empty`, `auto-award` |

At least 2 segments required, 3 recommended. All lowercase alphanumeric with single hyphens.

**Example**:
```typescript
import { docScreenshot } from "./utils/docScreenshot";

// After the component renders and assertions pass:
await docScreenshot(page, "badges--celebration-modal--auto-award");
```

**Rules**:
- Place `docScreenshot()` calls AFTER assertions that confirm the desired visual state
- The filename IS the contract between tests and docs — keep names stable
- Never manually edit files in `docs/assets/images/screenshots/auto/` — they are overwritten by CI
- Manually curated screenshots stay in `docs/assets/images/screenshots/` (parent directory)

#### Release Screenshots (Point-in-Time)

For release notes, use `releaseScreenshot` to capture screenshots that are **locked in amber** — they show the UI at a specific release and never change.

**Location**: `docs/assets/images/screenshots/releases/{version}/` (output)

```typescript
import { releaseScreenshot } from "./utils/docScreenshot";

await releaseScreenshot(page, "v3.0.0.b3", "landing-page", { fullPage: true });
```

**Key differences from `docScreenshot`:**
- Output: `docs/assets/images/screenshots/releases/{version}/{name}.png`
- **Write-once**: If the file already exists, the function is a no-op (won't overwrite)
- CI never touches the `releases/` directory
- Name is a simple kebab-case string (no `--` segment convention)
- Version must match `v{major}.{minor}.{patch}` format (with optional suffix)

**When to use which:**
- `docScreenshot` → README, quickstart, guides (always fresh)
- `releaseScreenshot` → Release notes (frozen at release time)

### Authenticated Playwright Testing (Live Frontend Debugging)

When you need to interact with the running frontend as an authenticated user (e.g., debugging why a query returns empty results), use Django admin session cookies to authenticate GraphQL requests.

**Architecture context**: The frontend uses Auth0 for authentication, but the Django backend also accepts session cookie auth. Apollo Client sends GraphQL requests directly to `http://localhost:8000/graphql/` (cross-origin from the Vite dev server at `localhost:5173`). Since `fetch` defaults to `credentials: 'same-origin'`, browser cookies aren't sent cross-origin. The workaround is to inject the session cookie into request headers via Playwright's route interception.

**Step 1: Set a password for the superuser** (one-time setup):
```bash
docker compose -f local.yml exec django python manage.py shell -c "
from django.contrib.auth import get_user_model
User = get_user_model()
u = User.objects.filter(is_superuser=True).first()
u.set_password('testpass123')
u.save()
print(f'Password set for {u.username}')
"
```

**Step 2: Playwright script pattern**:
```javascript
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext();
  const page = await context.newPage();

  // Collect console messages for debugging
  const consoleMsgs = [];
  page.on('console', msg => consoleMsgs.push('[' + msg.type() + '] ' + msg.text()));

  // 1. Login to Django admin to get session cookie
  await page.goto('http://localhost:8000/admin/login/');
  await page.fill('#id_username', '<superuser-username>');
  await page.fill('#id_password', 'testpass123');
  await page.click('input[type=submit]');
  await page.waitForTimeout(2000);

  // 2. Extract the session cookie
  const cookies = await context.cookies();
  const sessionCookie = cookies.find(c => c.name === 'sessionid');

  // 3. Intercept GraphQL requests to inject the session cookie
  //    (needed because Apollo sends cross-origin requests to :8000)
  await page.route('**/graphql/**', async (route) => {
    const headers = {
      ...route.request().headers(),
      'Cookie': 'sessionid=' + sessionCookie.value,
    };
    await route.continue({ headers });
  });

  // 4. Navigate to the frontend page under test
  await page.goto('http://localhost:5173/extracts');
  await page.waitForTimeout(5000);

  // 5. Inspect results
  const bodyText = await page.textContent('body');
  console.log(bodyText);

  await browser.close();
})();
```

**Run from the frontend directory** (where `playwright` is a dependency):
```bash
cd frontend && node /path/to/script.js
```

**Key details**:
- The Django admin login sets a `sessionid` cookie for `localhost`
- `page.route('**/graphql/**')` intercepts Apollo's requests to `localhost:8000/graphql/` and injects the cookie header
- The AuthGate will still show anonymous state (no Auth0 session), but GraphQL queries execute as the authenticated Django user
- This is useful for verifying backend query results, permission filtering, and data flow through the real frontend
- For verifying just the GraphQL response without the full frontend, `curl` with the session cookie also works:
  ```bash
  curl -s -X POST http://localhost:8000/graphql/ \
    -H "Content-Type: application/json" \
    -H "Cookie: sessionid=<session-key>" \
    -d '{"query":"query { extracts { edges { node { id name } } } }"}' | python3 -m json.tool
  ```

**Alternative — create a session programmatically** (no admin login needed):
```bash
docker compose -f local.yml exec django python manage.py shell -c "
from django.contrib.sessions.backends.db import SessionStore
from django.contrib.auth import get_user_model
User = get_user_model()
user = User.objects.filter(is_superuser=True).first()
session = SessionStore()
session['_auth_user_id'] = str(user.pk)
session['_auth_user_backend'] = 'django.contrib.auth.backends.ModelBackend'
session['_auth_user_hash'] = user.get_session_auth_hash()
session.save()
print(f'Session key: {session.session_key}')
"
```
Then use the printed session key directly in curl or Playwright route interception.

## Documentation Locations

- **Permissioning**: `docs/permissioning/consolidated_permissioning_guide.md`
- **Frontend Routing**: `docs/frontend/routing_system.md`
- **Frontend Auth Flow**: `docs/frontend/auth_flow.md`
- **PDF Data Layer**: `docs/architecture/PDF-data-layer.md`
- **PAWLs Format (v1/v2)**: `docs/architecture/pawls-format.md`
- **Parser Pipeline**: `docs/pipelines/pipeline_overview.md`
- **LLM Framework**: `docs/architecture/llms/README.md`
- **Collaboration System**: `docs/commenting_system/README.md`
- **Auth Pattern (detailed)**: `frontend/src/docs/AUTHENTICATION_PATTERN.md`
- **Documentation Screenshots**: `docs/development/screenshots.md`
- **Query Permission Patterns**: `docs/architecture/query_permission_patterns.md`
- **OS-Legal-Style Migration Guide**: `docs/frontend/os-legal-style-migration-guide.md`

## Branch Strategy

This project follows **trunk-based development**:

- Work directly on `main` branch or use short-lived feature branches
- Feature branches: `feature/description-issue-number`
- Merge feature branches quickly (within a day or two)
- Commit message format: Descriptive with issue references (e.g., "Closes #562")

## Changelog Maintenance

**IMPORTANT**: Always record significant changes — but **do NOT edit `CHANGELOG.md`
directly in a PR.** Add a *changelog fragment* under `changelog.d/` instead.

**Why:** every PR used to insert its entry at the top of `CHANGELOG.md`'s single
`## [Unreleased]` section, so concurrent PRs collided on the same lines and
produced perpetual merge conflicts (the file changed in ~hundreds of commits per
month). Each PR now adds its own uniquely-named file, so two PRs can never touch
the same lines — changelog merge conflicts become structurally impossible. The
fragments are collated into `CHANGELOG.md` at release time. (A `merge=union`
driver in `.gitattributes` is a safety net for any direct edit that slips
through; it auto-keeps both sides instead of conflicting.)

**How to add an entry** — create one file per change:

```
changelog.d/<slug>.<type>.md
```

- `<slug>` — anything unique; the PR number (`1901`) is ideal, or a short
  kebab-case description.
- `<type>` — one of `added`, `changed`, `deprecated`, `removed`, `fixed`,
  `security` (Keep a Changelog groups).
- **Body** = the markdown bullet(s) you'd have written under the section header,
  *without* the `### Fixed` header itself (the type comes from the filename).
- Keep the same quality bar: file paths, line numbers, issue/impact, rationale.
- Multiple categories in one PR → multiple fragments
  (`1908-search.added.md` + `1908-search.fixed.md`).

Tooling (`scripts/collate_changelog.py`): `--check` validates fragments (CI /
pre-commit), `--preview` prints the collated markdown, `--apply` folds fragments
into `CHANGELOG.md`'s `[Unreleased]` section and deletes them (release step).
See `changelog.d/README.md` for full details.

**When to add a fragment**:
- New features or models added
- Production code bugs fixed (document file location, line numbers, and impact)
- Breaking changes to APIs or data models
- Test suite fixes that reveal production issues
- Database migrations
- Architecture changes

**What to include**:
- File paths and line numbers for code changes
- Clear description of the issue and fix
- Impact on system behavior
- Migration notes if applicable

## Pre-commit Hooks

Automatically run on commit:
- black (Python formatting)
- isort (import sorting)
- flake8 (linting)
- prettier (frontend formatting)
- pyupgrade (Python syntax modernization)

Run manually: `pre-commit run --all-files`

## Common Pitfalls

1. **Frontend tests hanging**: Always use `--reporter=list` flag
2. **Permission N+1 queries**: Use `.visible_to_user()` NOT individual permission checks. For **Conversation list queries** in GraphQL resolvers, use `ConversationService` from `opencontractserver.conversations.services` - it provides request-level caching to avoid repeated corpus/document visibility subqueries. For annotation/relationship source privacy, looping `user_can` over rows dereferences `created_by_analysis` / `created_by_extract`; use list/queryset surfaces or `select_related("created_by_analysis", "created_by_extract")`.
3. **Missing GraphQL mocks**: Check variables match exactly (null vs undefined matters), add duplicates for refetches
4. **Notification duplication in tests**: Moderation methods auto-create ModerationAction records
5. **Structural annotation editing**: Always read-only except for superusers
6. **Missing signal imports**: Import signal handlers in `apps.py` `ready()` method
7. **PDF rendering slow in tests**: Increase timeouts to 20s+ for Chromium
8. **Cache serialization crashes**: Keep InMemoryCache definition inside wrapper, not test file
9. **Backend Tests Waiting > 10 seconds on Postgres to be Ready**: Docker network issue. Fix with: `docker compose -f test.yml down && docker kill $(docker ps -q) && docker compose -f test.yml down`
10. **Empty lists on direct navigation**: AuthGate pattern solves this (don't check auth status, it's always ready)
11. **URL desynchronization**: Use CentralRouteManager, don't bypass routing system
12. **Jotai state not updating**: Ensure atoms are properly imported and used with useAtom hook
13. **Writing sync agent tools**: All agent tools must be async. The `PydanticAIToolWrapper` accepts sync functions but calls them directly (no thread pool) — sync Django ORM calls will raise `SynchronousOnlyOperation`. Use the `a`-prefixed async versions in `core_tools.py`.
14. **PydanticAI `system_prompt` silently dropped**: When creating `PydanticAIAgent`, use `instructions=` NOT `system_prompt=`. The `system_prompt` parameter is only included when `message_history` is `None`, but OpenContracts' `chat()` persists a HUMAN message before calling pydantic-ai's `run()`, so history is always non-empty. This causes the system prompt to be silently dropped. See `docs/architecture/llms/README.md` for full details.
15. **Apollo cache `keyArgs` must use field argument names, not variable names**: In `cache.ts`, `relayStylePagination(["corpus", "name_Contains"])` uses the GraphQL **field argument** names (e.g., `extracts(corpus: $id, name_Contains: $text)`), NOT the variable names (`$id`, `$text`). Mismatched keyArgs silently fail to isolate cache entries, causing queries with different filters to share stale cached results.
16. **Playwright CT split-import rule**: In `frontend/tests/*.ct.tsx`, keep JSX-component imports in their own `import { Wrapper } from "./Wrapper"` statement, separate from any helper/constant imports from the same file. Playwright CT's babel transform only rewrites component references into `importRefs` when every specifier in the statement is a JSX component; mixing a component with helpers leaves the component unrewritten and `mount()` throws. Do not let auto-import organisers merge them.
17. **pydantic-ai dataclass-shape assertion at import time**: `opencontractserver/llms/history_processors.py` performs a module-level `hasattr(_msg_type, "__dataclass_fields__")` check on `ToolReturnPart`/`ModelRequest`/`ModelResponse` and raises `RuntimeError` if any of them stops being a stdlib `@dataclass`. The check is intentional — the in-run shrink helpers use `dataclasses.replace()` and the outer defensive try/except would otherwise swallow a silent shape change. The cost is that a routine `pip install --upgrade pydantic-ai` that flips one of these to a `BaseModel` will surface as a startup `RuntimeError` (`manage.py runserver`, Celery worker boot). The traceback names the offending message class — update the helpers in `history_processors.py` to use the new message API rather than disabling the assertion.
18. **Corrupted Docker iptables chains** (RARE): If you see `Chain 'DOCKER-ISOLATION-STAGE-2' does not exist` errors, Docker's iptables chains have been corrupted during docker cycling. Run this nuclear fix:
    ```bash
    sudo systemctl stop docker && sudo systemctl stop docker.socket && sudo ip link delete docker0 2>/dev/null || true && sudo iptables -t nat -F && sudo iptables -t nat -X && sudo iptables -t filter -F && sudo iptables -t filter -X 2>/dev/null || true && sudo iptables -t mangle -F && sudo iptables -t mangle -X && sudo iptables -t filter -N INPUT 2>/dev/null || true && sudo iptables -t filter -N FORWARD 2>/dev/null || true && sudo iptables -t filter -N OUTPUT 2>/dev/null || true && sudo iptables -P INPUT ACCEPT && sudo iptables -P FORWARD ACCEPT && sudo iptables -P OUTPUT ACCEPT && sudo systemctl start docker
    ```
    This completely resets Docker's networking and iptables state. Docker will recreate all required chains on startup.
19. **WebSocket 1011 reconnect churn (channels-redis ↔ redis-py `socket_timeout`)**: If every WebSocket consumer (`/ws/notification-updates/`, `/ws/agent-chat/`, …) drops with close code **1011** on a clean **~5-second beat** and the browser reconnects forever, it is almost certainly the channel layer's idle blocking read timing out *client-side*, NOT a frontend re-render/remount (the connect hook `frontend/src/hooks/useWebSocketAuth.ts` is re-render-safe: `url` is a value-compared string, callbacks are ref-held). channels-redis' receive loop issues a 5s server-side blocking pop (`bzpopmin`, `brpop_timeout=5`); **redis-py 8.0 changed the default `socket_timeout` from `None` to 5s**, so the client read deadline fires before the server's nil reply returns → `redis.exceptions.TimeoutError` escapes the consumer → Daphne emits 1011. `redis` is unpinned in `requirements/base.txt`, which is how it drifted to 8.0. **Fix (do NOT pin redis back below 8.0):** the `CHANNEL_LAYERS` host must be a dict with `socket_timeout: None` (see `config/settings/base.py` and `config/settings/test_integration.py`) — a bare `(host, port)` tuple or URL string inherits redis-py's default and reintroduces the bug. Server-side symptom in the Daphne logs: `Exception inside application: Timeout reading from redis:6379` immediately followed by `WSDISCONNECT`. Regression test: `test_redis_integration.py::TestChannelsRedisLayer::test_idle_receive_survives_blocking_pop_window`. See issue #1886.
20. **Reasoning models that require the OpenAI Responses API** (`gpt-5.6-sol` / `-terra` / `-luna`, and likely successors): two separate failures, both invisible on `/v1/chat/completions` and both fatal to an agent run.
    - *Function tools are rejected outright*: `Function tools with reasoning_effort are not supported for gpt-5.6-luna in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.` Every OpenContracts agent carries function tools, and zeroing reasoning discards the capability the model is chosen for, so the endpoint moves instead — `llms/model_factory.py::requires_responses_api` routes the family to `/v1/responses` on BOTH the DB-credential and env-credential paths. Routed automatically because these names are selectable in the System Settings LLM picker; without it, choosing one 400s every agent in the install with the reason visible only in a worker log.
    - *Orphaned `function_call` items*: the Responses API rejects a tool call whose paired `reasoning` item is missing — `Item 'fc_…' of type 'function_call' was provided without its required 'reasoning' item: 'rs_…'`. In-run compaction (`llms/history_processors.py`) drops `ThinkingPart`s from older messages, which Chat Completions tolerated for years. It now keeps reasoning on any response that also carries a `ToolCallPart`; reasoning-only responses still shed theirs. Expressed as an invariant, not a per-provider switch.
    - **Also register the context window** in `constants/context_guardrails.py`. There is no bare `gpt-5` entry to inherit, so an unlisted name falls to `DEFAULT_CONTEXT_WINDOW` (128K) and sizes a 1M-window model at an eighth of its budget — which moves when a deep-research run compacts, not just a reported number. Same trap as the `gpt-4.1` entries above it.
21. **`tool_call_log` does not see the whole toolset by default**: `research_tasks.py::_audited` wraps only the closures that module builds. Everything from the corpus agent's *default* toolset (`similarity_search`, `list_documents`, `ask_document`, …) bypasses it unless `_audit_default_toolset` runs, and that helper reaches into pydantic-ai's `Tool.function_schema.function` seam — which a framework upgrade could move. **Never read a tool's absence off this log without corroboration.** Doing so once produced a confidently wrong conclusion ("ten runs never searched by meaning") that drove a prompt rewrite; the agent had been calling `similarity_search` the whole time. Independent check that costs nothing: a phrase lookup (`find_citable_passages` → `AnnotationService.search_corpus_annotation_text`) performs a TEXT query and embeds nothing, while a semantic search must embed its query — so `Embedding text with MicroserviceEmbedder` in the worker log during a run is proof of semantic retrieval regardless of what the audit log shows. Note also that closures passed to the factory are RE-WRAPPED by it, so marker attributes set on our wrapper are not reachable from the resolved toolset; `_audit_default_toolset` therefore skips by NAME.