agent-framework / azure-contentunderstanding

# AGENTS.md — azure-contentunderstanding

## Package Overview

`agent-framework-azure-contentunderstanding` integrates Azure Content Understanding (CU)
into the Agent Framework as a context provider. It automatically analyzes file attachments
(documents, images, audio, video) and injects structured results into the LLM context.

## Public API

| Symbol | Type | Description |
|--------|------|-------------|
| `ContentUnderstandingContextProvider` | class | Main context provider — extends `ContextProvider` |
| `AnalysisSection` | enum | Output section selector (MARKDOWN, FIELDS, etc.) |
| `DocumentStatus` | enum | Document lifecycle state (ANALYZING, UPLOADING, READY, FAILED) |
| `FileSearchBackend` | ABC | Abstract vector store file operations interface |
| `FileSearchConfig` | dataclass | Configuration for CU + vector store RAG mode |

## Architecture

- **`_context_provider.py`** — Main provider implementation. Overrides `before_run()` to detect
  file attachments, call the CU API, manage session state with multi-document tracking,
  and auto-register retrieval tools for follow-up turns.
  - **Analyzer auto-detection** — When `analyzer_id=None` (default), `_resolve_analyzer_id()`
    selects the CU analyzer based on media type prefix: `audio/` → `prebuilt-audioSearch`,
    `video/` → `prebuilt-videoSearch`, everything else → `prebuilt-documentSearch`.
  - **Multi-segment output** — CU splits long video/audio into multiple scene segments
    (each a separate `contents[]` entry with its own `startTimeMs`, `endTimeMs`, `markdown`,
    and `fields`). `_extract_sections()` produces:
    - `segments`: list of per-segment dicts, each with `markdown`, `fields`, `start_time_s`, `end_time_s`
    - `markdown`: concatenated at top level with `---` separators (for file_search uploads)
    - `duration_seconds`: computed from global `min(startTimeMs)` → `max(endTimeMs)`
    - Metadata (`kind`, `resolution`): taken from the first segment
  - **Speaker diarization (not identification)** — CU transcripts label speakers as
    `<Speaker 1>`, `<Speaker 2>`, etc. CU does **not** identify speakers by name.
  - **file_search RAG** — When `FileSearchConfig` is provided, CU-extracted markdown is
    uploaded to an OpenAI vector store and a `file_search` tool is registered on the context
    instead of injecting the full document content. This enables token-efficient retrieval
    for large documents.
- **`_models.py`** — `AnalysisSection` enum, `DocumentStatus` enum, `DocumentEntry` TypedDict,
  `FileSearchConfig` dataclass.
- **`_file_search.py`** — `FileSearchBackend` ABC, `OpenAIFileSearchBackend`,
  `FoundryFileSearchBackend`.

## Key Patterns

- Follows the Azure AI Search context provider pattern (same lifecycle, config style).
- Uses provider-scoped `state` dict for multi-document tracking across turns.
- Auto-registers `list_documents()` tool via `context.extend_tools()`.
- Configurable timeout (`max_wait`) with `asyncio.create_task()` background fallback.
- Strips supported binary attachments from `input_messages` to prevent LLM API errors.
- Explicit `analyzer_id` always overrides auto-detection (user preference wins).
- Vector store resources are cleaned up in `close()` / `__aexit__`.

## Samples

| Sample | Description |
|--------|-------------|
| `samples/02-agents/context_providers/azure_content_understanding/01_document_qa.py` | Upload a PDF via URL, ask questions about it |
| `samples/02-agents/context_providers/azure_content_understanding/02_multi_turn_session.py` | AgentSession persistence across turns |
| `samples/02-agents/context_providers/azure_content_understanding/03_multimodal_chat.py` | PDF + audio + video parallel analysis |
| `samples/02-agents/context_providers/azure_content_understanding/04_invoice_processing.py` | Structured field extraction with `prebuilt-invoice` analyzer |
| `samples/02-agents/context_providers/azure_content_understanding/05_large_doc_file_search.py` | CU extraction + OpenAI vector store RAG |
| `samples/02-agents/devui/agent_content_understanding/` | DevUI web UI for CU-powered chat |
| `samples/02-agents/devui/agent_content_understanding_file_search_*/` | DevUI web UI combining CU + file_search RAG |

## Running Tests

```bash
uv run poe test -P azure-contentunderstanding
```