gemini-pdf

---
name: gemini-pdf
description: Convert PDF files to clean Markdown using Antigravity CLI (`agy`). Best for academic papers with math, tables, and structured content. The native pipeline feeds page-range chunk PDFs directly to Gemini's vision pathway (agy reads the PDF itself — no lossy text extraction), with an automatic quality-gated fallback to a Mathpix hybrid and Mathpix standalone. Zero marginal cost via a Google subscription. Triggers include "convert PDF to markdown with Antigravity", "agy PDF conversion", "PDF to markdown with math", and "convert paper to markdown".
---

# PDF-to-Markdown Conversion (via Antigravity CLI)

> **Note**: The skill folder is named `gemini-pdf` for backward compatibility with the `/gemini-pdf` slash command. Internally it drives Antigravity CLI (`agy`), Google's successor to the original Gemini CLI.

## When to Use

- Academic papers with mathematical notation (LaTeX reconstructed from the rendered page)
- Papers with complex tables, theorems, proofs, or two-column layouts
- Scanned PDFs (the native vision pathway OCRs them; the Mathpix fallback also handles them)
- Zero-cost conversion via a Google One AI subscription (no API keys for the primary path)

## When NOT to Use

- **Need exact LaTeX source** → ask the author or use Mathpix directly
- **Enormous documents (>500 pages)** → split into chapters first (hard page ceiling)

## Locating the script

Examples below call the script through `$GEMINI_PDF`. Set it once per shell so they work whether the skill was installed as a plugin or copied manually:

```bash
if [ -n "$CLAUDE_PLUGIN_ROOT" ]; then
  GEMINI_PDF="$CLAUDE_PLUGIN_ROOT/skills/gemini-pdf"   # installed as a Claude Code plugin
else
  GEMINI_PDF="$HOME/.claude/skills/gemini-pdf"          # installed by manual copy
fi
```

(The script self-resolves its own helper scripts, so it runs correctly from any location.)

## Quick Usage

```bash
# Basic: native pipeline, outputs paper.md alongside paper.pdf
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf

# Specify output path
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf /tmp/output.md

# Choose a pipeline explicitly
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --pipeline text
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --hybrid
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --force-mathpix

# Disable auto-chunking (single agy pass over the whole PDF)
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --no-chunk

# Disable the quality-gated fallback chain
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --no-fallback

# Custom quality threshold (default: 60)
bash "$GEMINI_PDF/scripts/pdf_to_markdown.sh" paper.pdf --quality-threshold 40
```

## Architecture

### Native pipeline (default)

`agy` reads the PDF **directly** through Gemini's vision/document pathway — math layout, column order, and table geometry are seen as rendered, not guessed back from lossy text extraction:

```
pdfinfo → page count P
P ≤ 10 → single chunk (whole PDF)
P > 10 → split_pdf_pages.py → chunk PDFs of ≤ 10 pages (1-indexed inclusive ranges)
for each chunk SEQUENTIALLY (global agy lock):
    isolated temp dir containing only that chunk PDF
    agy --model "$MODEL" --add-dir <dir> --print-timeout <T>s -p "<prompt>"
    validate (nonce end-sentinel + ratio guards) → retry → bisect on repeated failure
concatenate → quality gates → sanitize → output.md
```

- The first chunk's prompt produces YAML front matter (`title/authors/year/journal/doi`) **plus** the `# Title` heading; continuation chunks follow a seam protocol (transcribe from the first character of the page, resume exactly after the previous chunk's tail — provided as read-only context — never skip, never repeat).
- Failed/truncated chunks are retried once, then recursively bisected down to single pages before a `<!-- CHUNK N FAILED -->` placeholder is accepted.
- Prompts instruct the model to flag unreadable passages as `<!-- UNCERTAIN: ... -->` rather than guess — a flagged gap always beats fluent fabrication.

### Concurrency constraint (hard, probe-verified)

Two concurrent `agy --print` processes **hang** (0 bytes output, both ignore their own `--print-timeout`; verified live 2026-07-07 on agy 1.0.16, plain and with `--new-project`). Consequently:

- All agy calls run strictly sequentially under a global lock (`$TMPDIR/gemini-pdf-agy.lock`); concurrent script invocations serialize automatically.
- Every agy call is wrapped in an external watchdog (`TIMEOUT+60s`) that kills the **whole process group** and sweeps orphans — agy's own timeout cannot be trusted under contention.
- `agy --sandbox` also hangs the native read (probe-verified) and is not used.

### Fallback chain (auto, quality-gated)

```
native ──eligible & ≥threshold──→ done
   │ low
   ▼
hybrid: mathpix --mode extract → agy structural cleanup (math preserved verbatim)
   │ low
   ▼
mathpix standalone → best eligible attempt wins
```

Winner selection is two-stage: **eligibility gates** first (no failed chunks, fidelity floors, completeness floor), then comparison on **literalness-neutral** dimensions only (math quality, structure, OCR, failed chunks) — recall-vs-source dimensions would unfairly reward the most literal attempt. A later stage must beat an earlier one by ≥5 points to displace it. Losing attempts are kept as `output.attempt-<stage>.md`.

Degraded environments: agy missing → Mathpix standalone (the text pipeline needs agy too); poppler missing → text pipeline with markitdown; pypdf missing → poppler page splitter.

### Model configuration

`agy` ≥ 1.0.5 has a working `--model` flag; the script passes `GEMINI_PDF_MODEL` (default **"Gemini 3.1 Pro (High)"**) on every call and falls back to the `~/.gemini/antigravity-cli/settings.json` model only if the installed agy lacks the flag. List models with `agy models`.

### Quality verification

`scripts/quality_check.py` scores 0-100 with per-dimension breakdown:

| Dimension | Weight (with source text) | Checks |
|-----------|--------------------------|--------|
| Content completeness | 20% | raw output vs source-text ratio (band 0.5–2.0) |
| Math quality | 20% | balanced `$`/`$$` (currency-aware), well-formed LaTeX |
| Structural integrity | 15% | title, sections, bibliography, tables, theorem markers |
| OCR artifacts | 10% | replacement chars, garbled runs, ligatures |
| Failed chunks | 10% | any failure hard-caps the overall score at ≤50 |
| Numeric fidelity | 15% | recall of source numerals (boilerplate/page-number filtered) |
| Trigram recall | 10% | recall of source prose trigrams (normalized) |

Plus: `comparison_score` (literalness-neutral, for cross-attempt selection), `eligible` + reasons, UNCERTAIN-marker count (small penalty), and front-matter detection. Corrupted PDF text layers (some 1990s journal PDFs encode `.` as `+` or `(` as `Ž`) are detected automatically and the source-dependent dimensions are skipped — recall against a garbled reference would punish a *correct* conversion. Without source text, a legacy five-dimension weighting applies. Non-mathematical documents redistribute the math weight adaptively.

### Sanitization

Converted Markdown originates from an untrusted PDF; `scripts/sanitize_md.py` strips `<script>`/`<iframe>`/`<object>` blocks, HTML event-handler attributes, and `javascript:` URLs before any downstream use, reporting every removal.

### Optional reference-library sync

**Opt-in and off by default.** When `GEMINI_PDF_REFERENCE_DIR` points to an existing directory (layout: `pdf/`, `md/`, `reference.bib`), the final winner (only — never intermediate attempts) is synced there: with `--bib-key KEY`, PDF → `pdf/KEY.pdf`, MD → `md/KEY.md`, and a BibTeX entry is appended (metadata precedence: JSON sidecar → the MD's own YAML front matter → key-only entry). Copies are skip-if-exists; `--force-sync` overwrites the PDF/MD copies (bib entries are never replaced). Without the env var set, nothing is written outside your output path. See [docs/gemini-pdf.md](../../docs/gemini-pdf.md).

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `GEMINI_PDF_MODEL` | `Gemini 3.1 Pro (High)` | Passed to `agy --model` on every call |
| `GEMINI_PDF_TIMEOUT` | `600` | Per-chunk timeout (s); watchdog fires at +60s |
| `GEMINI_PDF_PIPELINE` | `native` | `native` \| `text` \| `hybrid` \| `mathpix` |
| `GEMINI_PDF_PAGES_PER_CHUNK` | `10` | Native pipeline pages per chunk |
| `GEMINI_PDF_MAX_PAGES` | `500` | Hard page ceiling (split larger docs first) |
| `GEMINI_PDF_CHUNK_THRESHOLD` | `50000` | Text pipeline: chunk above this many bytes |
| `GEMINI_PDF_CHUNK_MAX` | `100000` | Text pipeline: max bytes per chunk |
| `GEMINI_PDF_QUALITY_THRESHOLD` | `60` | Quality gate for the fallback chain |
| `GEMINI_PDF_MATHPIX_FALLBACK` | `true` | Enable/disable automatic Mathpix fallback |
| `GEMINI_PDF_LOCK_DIR` | `$TMPDIR/gemini-pdf-agy.lock` | Global agy serialization lock |
| `GEMINI_PDF_REFERENCE_DIR` | (unset — opt-in) | **Optional.** Central reference library root; sync is skipped entirely unless set to an existing directory |
| `MATHPIX_APP_ID` / `MATHPIX_APP_KEY` | (none) | Mathpix API credentials (hybrid + fallback) |

## Flags

| Flag | Description |
|------|-------------|
| `--pipeline native\|text\|hybrid\|mathpix` | Choose the conversion pipeline |
| `--extractor pdftotext\|markitdown` | Text-pipeline extractor (implies `--pipeline text`) |
| `--no-chunk` | Single agy pass (native: whole PDF; text: no splitting) |
| `--no-fallback` | Disable the quality-gated fallback chain |
| `--force-mathpix` | Mathpix standalone (skip agy entirely) |
| `--hybrid` | Mathpix extraction + agy structural cleanup |
| `--bib-key KEY` | Sync winner to the reference library under KEY (needs `GEMINI_PDF_REFERENCE_DIR`) |
| `--force-sync` | Overwrite existing library PDF/MD copies |
| `--quality-threshold N` | Override quality gate (default 60) |

## Output files

For `output.md` the script may also write: `output.err` (agy stderr per chunk), `output.meta.json` (per-chunk ledger: page ranges, bytes, retries, status), `output.quality.json` (full quality report), and `output.attempt-<stage>.md` (losing fallback attempts, kept for manual override).

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `agy not found` | `curl -fsSL https://antigravity.google/cli/install.sh \| bash` |
| `Authentication required` from `agy` | Run `agy` interactively once and complete the OAuth flow |
| Conversion very slow | Chunks run sequentially by design (parallel agy hangs — verified). ~1-4 min/chunk is normal |
| Another conversion seems stuck | Check the lock: `ls $TMPDIR/gemini-pdf-agy.lock` — stale locks from dead processes are removed automatically |
| Timeout on a dense chunk | Increase `GEMINI_PDF_TIMEOUT=1200`; the pipeline also auto-bisects failing chunks |
| `<!-- CHUNK N FAILED -->` in output | That page range failed after retries + bisection; re-run, or convert those pages separately |
| Math still garbled | Confirm the model: `GEMINI_PDF_MODEL="Gemini 3.1 Pro (High)"`; try `--hybrid` (Mathpix math OCR) |
| Scanned PDF (no text layer) | Native pipeline handles it (vision); `--force-mathpix` is the alternative |
| Low quality score | Inspect `output.quality.json`; try `--hybrid` or `--force-mathpix`; lower `--quality-threshold` |
| Mathpix fallback not triggering | Set `MATHPIX_APP_ID` and `MATHPIX_APP_KEY` |
| Empty output | Check `output.err` for agy errors |
| Wrong model used | This agy build may lack `--model` (warning is printed); set it in `~/.gemini/antigravity-cli/settings.json` |