airweave / rules

---
globs: **/fern/**
alwaysApply: false
---
# Fern Documentation Generation System - Internal Guide

## Overview
The custom internal Fern documentation generator automatically creates MDX documentation for Airweave connectors by introspecting Python source code using AST (Abstract Syntax Tree) parsing.

## Architecture Flow

### 1. Discovery Phase
**Entry:** `update_connector_docs/__main__.py:main()`
- Scans `backend/airweave/platform/sources/` directory
- Identifies all `.py` files as potential connectors
- Example: `asana.py`, `slack.py`, `google_drive.py`

### 2. Parsing Phase

#### Source Parser (`source_parser.py`)
Extracts metadata from `@source` decorator:
```python
@source(
    name="Asana",                    # Display name
    short_name="asana",              # File/URL identifier
    auth_methods=[...],              # Auth methods list
    oauth_type=OAuthType.WITH_REFRESH,  # OAuth type enum
    auth_config_class="AsanaConfig", # String reference to config
    config_class="AsanaConfig",      # Source-specific config
    labels=["Project Management"]    # Categories
)
```

**AST Extraction Process:**
1. Parse decorator arguments positionally and by keyword
2. Fall back to class attributes (`_auth_type`, `_config_class`)
3. Last resort: regex pattern matching in raw source

#### Entity Parser (`entity_parser.py`)
- Scans `platform/entities/{connector_name}.py`
- Extracts entity classes and their field definitions
- Builds entity hierarchy for documentation

#### Auth/Config Parsers
- `auth_parser.py`: Maps auth config class names to their field definitions
- `config_parser.py`: Maps source config classes to their field requirements

### 3. Generation Phase

#### MDX Generator (`mdx_generator.py`)
Creates structured MDX with:
- **Header**: Icon + connector name
- **Configuration**: Source docstring description
- **Authentication**:
  - OAuth flows (managed vs BYOC)
  - Direct auth field requirements
- **Source Config**: Additional configuration fields
- **Entity Schema**: Data structure documentation

### 4. Output Phase
- Creates/updates `fern/docs/pages/connectors/{name}/main.mdx`
- Copies icon from `frontend/public/icons/`
- Updates `docs.yml` navigation structure

## Key Components

### Decorator Metadata (`platform/decorators.py`)
The `@source` decorator sets class attributes:
- `_is_source = True` (marker)
- `_name`, `_short_name` (identification)
- `_auth_methods`, `_oauth_type` (auth config)
- `_auth_config_class`, `_config_class` (config references)

### AST Parsing Strategy
1. **Direct decorator parsing**: Extract from `ast.Call` nodes
2. **Class attribute fallback**: Check `ast.Assign` nodes
3. **Regex fallback**: Pattern match in raw source text

### MDX Special Handling
- Escapes `<` and `>` to prevent JSX interpretation
- Uses HTML entities for safe rendering
- Preserves curly braces using string concatenation

## Configuration Resolution

### OAuth Detection
- Check `oauth_type` field presence
- Determine BYOC by inheritance chain:
  - Direct: `OAuth2BYOCAuthConfig`
  - Inherited: Parent/grandparent check

### Field Documentation
- Primary: Field's own description
- Fallback: Parent class field description
- Default: "No description"

## File Structure
```
fern/
├── scripts/
│   └── update_connector_docs/
│       ├── __main__.py         # Entry point
│       ├── constants.py        # Path definitions
│       ├── parsers/
│       │   ├── source_parser.py    # AST parsing
│       │   ├── entity_parser.py    # Entity extraction
│       │   ├── auth_parser.py      # Auth config mapping
│       │   └── config_parser.py    # Source config mapping
│       ├── generators/
│       │   └── mdx_generator.py    # MDX creation
│       └── utils/
│           └── file_utils.py       # File operations
└── docs/
    └── pages/
        └── connectors/
            └── {connector_name}/
                ├── icon.svg
                └── main.mdx
```

## Execution
```bash
cd fern/scripts
python update_connector_docs.py
```

Processes all sources, generates MDX, updates navigation.