Knowledge Ingestion Planner

Purpose

Design a knowledge ingestion pipeline: how documents enter the system, are parsed, enriched, chunked, and indexed with reliable incremental updates.

Input: Source types (PDF, wiki, API, DB), volume, update frequency, access control requirements
Output: Ingestion Pipeline Specification with connector map, parsing rules, metadata schema, and sync strategy
Examples: See examples.md for worked input/output.

Workflow

Step 1: Inventory sources

Identify authoritative vs supplemental sources; flag duplicates across sources.

Step 2: Design connector layer

Each connector outputs normalized RawDocument with provenance:

source_id, uri, mime, fetched_at, acl_principal, raw_bytes

Step 3: Define parsing and extraction

Document fallback when parsing confidence is low (quarantine queue).

Step 4: Metadata enrichment

Required metadata per chunk parent document:

Optional: auto-summary, entity extraction, section headings.

Step 5: Hand off to chunking and indexing

Define contract with chunking-strategy-advisor:

• Send cleaned markdown + structure hints (headings, tables)
• Chunk size/overlap per doc type
• Parent-child chunk linking for citations

Indexing pipeline:

Parse → Enrich → Chunk → Embed → Upsert vector store → Verify count

Step 6: Incremental sync and deletion

Idempotency key: source_id + content_hash.

Step 7: Validate

Run Validation checklist.

Decision Rules

Validation

• Source inventory with volume and update frequency
• Connector pattern per source
• Parsing rules per MIME type
• Metadata schema with ACL fields
• Chunking handoff documented
• Incremental sync and delete behavior defined
• Idempotency and reconciliation plan
• Ingestion SLA and error quarantine queue

Anti-patterns

• Reindex everything nightly — wasteful at scale; misses true incremental.
• Ignore ACL at ingest — retrieval leaks across tenants.
• Chunk before clean — boilerplate pollutes every chunk.
• No deletion path — stale content forever in answers.
• Single parser for all PDFs — table-heavy docs need layout models.

Best Practices

• Store raw originals in object storage; derived chunks reproducible.
• Log ingest lineage: chunk → doc → source fetch timestamp.
• Sample 1% of ingested docs for human QA each week.
• Pair with embedding-strategy-advisor for model/version pinning.
• Monitor ingest lag and embedding failure rate.

Output Structure

# Ingestion Pipeline: [Knowledge Base Name]

## Sources
| Source | Connector | Schedule | ACL |
|--------|-----------|----------|-----|

## Architecture
[Diagram: sources → queue → parse → enrich → chunk → index]

## Parsing Rules
| MIME | Parser | Fallback |
|------|--------|----------|

## Metadata Schema
[Field definitions]

## Sync Strategy
[Incremental rules]

## SLAs & Monitoring
| Metric | Target |
|--------|--------|

## Error Handling
[Quarantine, retry]

Next Skills