Document Embeddings

Pipeline: extract → chunk → embed → upsert to Qdrant.

Prerequisites

• llama-embedding (from llama.cpp) — for creating embeddings
• pdftotext (from poppler) — for PDF extraction
• jq — for JSON construction
• curl — for Qdrant REST API calls
• Qdrant running locally (default: http://localhost:6333)

The embedding model is downloaded automatically from Hugging Face on first run.

Usage

scripts/pipeline.sh -d <document> [options]

Required Arguments

Optional Arguments

Examples

# Embed a single PDF, default model (nomic-embed-text-v1.5 Q4_K_M)
scripts/pipeline.sh -d report.pdf

# Embed a directory of Markdown files
scripts/pipeline.sh -d ./docs/

# Embed with custom chunking, no full text stored
scripts/pipeline.sh -d notes.pdf -s 300 -o 30 -n

# Embed with GPU offload, delete old points first
scripts/pipeline.sh -d data.json -g all --delete

Supported File Types

Directories are supported — all files are processed recursively.

How It Works

1. Extract — PDFs are converted with pdftotext; all other files are read as raw text.
2. Chunk — Text is split into word-based chunks (default 500 words) with configurable overlap (default 50 words) using a sliding window.
3. Embed — Each chunk is passed to llama-embedding via -hf (downloads/caches from Hugging Face automatically).
4. Upsert — Embeddings are batched (64 per request) and uploaded to Qdrant via its REST API.

Model

The default model is nomic-embed-text-v1.5 (Q4_K_M quantization, 768-dim vectors). It is downloaded on first use and cached by llama.cpp — subsequent runs use the local copy.

To use a different model:

scripts/pipeline.sh -d doc.pdf --hf nomic-ai/nomic-embed-text-v1.5-GGUF:Q8_0

Collection Naming

The Qdrant collection name is derived from the HuggingFace repo:

• nomic-ai/nomic-embed-text-v1.5-GGUF:Q4_K_M → nomic-ai-nomic-embed-text-v1.5-gguf-q4-k-m

Payload Structure

With full text (default):

{
  "source_file": "report.pdf",
  "chunk_index": 3,
  "text": "The full chunk content here...",
  "word_range": [1400, 1850]
}

Without full text (-n flag):

{
  "source_file": "report.pdf",
  "chunk_index": 3,
  "word_range": [1400, 1850]
}

Point IDs are deterministic (SHA-256 of source_file + chunk_index), making re-ingestion idempotent.

Performance Notes

• The model downloads on first use (~200MB for Q4_K_M) and is cached locally by llama.cpp.
• For very large documents (1000+ chunks), consider running llama-server for faster batched embedding.

Errors

The script validates:

• Document/file exists
• Qdrant is reachable at the specified URL
• Embedding output is valid JSON