Literature Discovery

Search academic databases, curate relevant papers, and populate a project's docs/literature.md, bib file, and PDF collection.

Arguments

• /literature [project-slug] — run full pipeline for a project
• /literature [project-slug] --refresh — re-run searches, merge with existing candidates
• /literature [project-slug] --fetch-pdfs — only download PDFs for already-curated papers
• /literature [project-slug] --cite-graph DOI — expand candidates via citation graph of a specific paper
• /literature [project-slug] --audit — audit an existing bib for hallucinated or wrong entries (see Audit mode below). Reports only; never edits.
• /literature [project-slug] --update <citekey> [--artifact <build/path>] — surgical single-entry edit. Update one literature entry's annotation (key findings, relevance note, supersession marker) when a /next iteration produced a result that connects to or contradicts that paper. Reads only CLAUDE.md, literature.md, paper/*.bib, and the optional triggering artifact. Does not re-run searches, does not re-walk the bibliography, does not re-curate. Use from /next step 5 when a single citation needs a note added or revised. The <citekey> is the BibTeX key as used in literature.md and paper/*.bib.

If no project slug is given, infer from the current working directory.

Pipeline overview

1. Read context  →  2. Generate queries  →  3. API search
                                                  ↓
4. First curation (Claude)  →  5. Citation graph on key papers  →  6. Second curation
                                                                          ↓
7. Write literature.md + bib  →  8. Download PDFs

Step-by-step

1. Find workspace root and project

Locate $ROOT by searching upward for CLAUDE.md next to projects/ and pipelines/. Resolve the project path: $ROOT/projects/{slug}/.

Read these files to build project context:

• docs/summary.md (required)
• docs/thinking.md
• docs/methods.md
• docs/hypotheses.md
• docs/institutions.md
• docs/literature.md (existing entries)
• paper/*.tex (if exists, scan for \cite{} keys)
• paper/references.bib or paper/library.bib (existing bib entries)

2. Generate search queries

From the project context, generate 10–20 search queries covering:

• The core research question
• The identification strategy / methods
• The institutional setting
• Key theoretical mechanisms
• Adjacent literatures that the paper should cite or position against

Save queries to docs/literature/queries.json as a JSON array of strings.

3. Run API searches

python3 $ROOT/research-kit/tools/literature_search.py search \
    --queries $PROJECT/docs/literature/queries.json \
    --outdir $PROJECT/docs/literature

Saves:

• docs/literature/candidates.json — merged, deduplicated candidates with abstracts
• docs/literature/search_log.json — queries, timestamps, result counts

The script merges with any existing candidates.json, so re-running is safe.

4. First curation (Claude)

Read candidates.json. Skim titles and abstracts. Identify:

• The 20–60 genuinely relevant papers
• The 5–10 most central papers (for citation graph expansion)

Important: Do NOT invent or hallucinate paper details. Every paper in the final output must come from candidates.json (i.e., from an API response).

5. Citation graph expansion on key papers

For the 5–10 most central papers identified in step 4, expand via citation graphs. Pass multiple DOIs comma-separated:

python3 $ROOT/research-kit/tools/literature_search.py cite-graph \
    --doi 10.xxxx/yyyy,10.aaaa/bbbb,10.cccc/dddd \
    --outdir $PROJECT/docs/literature

This fetches papers that cite and are cited by each key paper, and merges them into candidates.json.

6. Second curation (Claude)

Re-read the now-expanded candidates.json. Check the newly added papers from citation graphs for relevance. This catches important papers that keyword search missed but are one hop from the core literature.

7. Write literature.md and bib entries

Update docs/literature.md following the project docs contract format. Every bullet must begin with a [cite:<bibkey>] token — this is what makes the entry addressable from other docs (per rules/citations.md). Without it, anything written elsewhere as [cite:<bibkey>] will fall back to literal text and not link to the index. Include a file: field for each paper so the PDF is easy to locate:

## Topic heading

- [cite:avis2018government] Author (Year) "Title"
  - topic: what the paper is about
  - relevance: why it matters for this project
  - file: citekey.pdf

The file: field, the [cite:...] key, and the BibTeX key must all be the same citekey. This ensures PDFs, bib entries, literature.md, and inline citations across all docs resolve to the same identifier — and the site build can deep-link to the bullet via literature/index.html#cite-<bibkey>.

Citekey convention

Citekeys follow the format {author}{year}{word}, all lowercase, no separators:

• {author} — first author's last name only. No EtAl, no coauthor names. Hyphenated names collapse the hyphen: Genicolo-Martins → genicolomartins. Particles stay attached: Della Porta → dellaporta, Di Tella → ditella.
• {year} — four-digit publication year.
• {word} — first substantive word of the title (lowercase). Skip articles and prepositions: a, an, the, of, in, on, for, and, to, do, how, is, are, as. Examples: "The Spoils of Victory" → spoils; "Do Government Audits Reduce Corruption?" → government.
• Institutional authors: lowercase the institution: oecd2016preventing, worldbank2020procurement.
• Disambiguation: if two papers produce the same key (same first author, year, and first substantive word), use the second substantive title word instead. If still ambiguous, concatenate the first two: decarolis2022rules vs decarolis2022corruption.

For bib entries:

• Prefer metadata from Crossref (run crossref-verify if needed)
• If Crossref is unavailable, use API metadata but flag as unverified
• Add new entries to existing paper/references.bib or paper/library.bib
• Do NOT overwrite existing entries — only append new ones

8. Create PDF manifest and download

After writing literature.md and bib entries, create a manifest file at docs/literature/manifest.json — a JSON array of objects with:

[
  {
    "citekey": "avis2018government",
    "title": "Do Government Audits Reduce Corruption?",
    "doi": "10.1086/699209",
    "oa_pdf_url": "https://..."
  }
]

The citekey must match the bib entry key exactly. To populate oa_pdf_url, look up each paper's DOI in candidates.json for the OA URL. Also populate doi for every paper — look up DOIs manually for papers where candidates.json has none (working papers, older papers, etc.).

8a. Check existing PDFs first

Before downloading anything, check if PDFs already exist locally. Many papers will already be in the shared literature folder or other project folders:

# Check shared literature folder
ls ~/Dropbox/literature/ | grep -i {author-name}

# Check other project literature folders
find ~/Dropbox/*/literature/ -iname "*{author}*" 2>/dev/null

Copy any matches to the project's literature folder with the correct citekey name:

cp ~/Dropbox/literature/"Author - Year - Title.pdf" ~/Dropbox/{project-folder}/literature/{citekey}.pdf

8b. Download OA and preprint PDFs

Determine the Dropbox path for the project:

ls ~/Dropbox/ | grep -i {slug-or-keywords}

Create literature/ subdirectory if needed. Then run:

python3 $ROOT/research-kit/tools/literature_search.py fetch-pdfs \
    --manifest $PROJECT/docs/literature/manifest.json \
    --outdir ~/Dropbox/{project-folder}/literature

This downloads OA PDFs and saves paywalled URLs. PDFs are named {citekey}.pdf — matching the bib key and the file: field in literature.md.

8c. Download paywalled PDFs (requires campus network)

Programmatic HTTP clients (Python requests, curl, wget, headless browsers) cannot download paywalled PDFs — publishers use bot detection that blocks everything except real browser sessions, even from authorized university IPs.

The only reliable method is to use the user's actual Chrome browser. To do this:

1. Set Chrome to auto-download PDFs via system policy (survives restarts, takes effect immediately after policy reload):

   sudo mkdir -p /etc/opt/chrome/policies/managed
   echo '{"AlwaysOpenPdfExternally":true,"DownloadDirectory":"/path/to/literature/folder"}' \
     | sudo tee /etc/opt/chrome/policies/managed/download_pdf.json

   The user needs to run this with sudo (use ! sudo ... from the prompt). Verify at chrome://policy → click "Reload policies".

2. Generate an HTML download page at /tmp/download_papers.html that uses <a target="_blank"> clicks to open each PDF URL. With the policy active, Chrome downloads the PDF instead of displaying it. Use DOI URLs (https://doi.org/...) for publisher papers — direct PDF URLs often 404. Use direct links only for NBER, SSRN, arXiv preprints.

   Structure the page with a "Download All" button that clicks each link sequentially with ~4s delays. Group papers into:

   • Direct PDF links (NBER, SSRN, arXiv) — auto-download
   • DOI links (publisher articles) — opens article page, user clicks PDF button
   • Books/working papers without URLs — skip
3. Rename downloaded files after the user confirms downloads are complete. Publishers use their own filenames (e.g., w31266.pdf, rest_a_01101.pdf, 1-s2.0-S030440762030378X-main.pdf). Write a rename script that maps publisher filenames to citekeys.
4. Clean up: Remove the Chrome policy and restore settings when done:

   sudo rm /etc/opt/chrome/policies/managed/download_pdf.json

If the user is not on campus, skip paywalled downloads and report which papers need to be downloaded later from the campus network.

Report to the user:

• How many PDFs were found locally (from shared folders)
• How many OA PDFs were downloaded
• How many paywalled PDFs were downloaded via browser
• How many are still missing (with reasons: book, no URL, no access)

Directory structure

After running, the project should have:

project/
  docs/
    literature.md                        # curated, annotated (with file: fields)
    literature/
      queries.json                       # search queries used
      candidates.json                    # all API results merged + deduplicated
      search_log.json                    # timestamps and counts
      manifest.json                      # curated papers with citekeys + OA URLs
  paper/
    references.bib or library.bib        # bib entries (existing + new)

And in Dropbox:

~/Dropbox/{project-folder}/literature/
  {citekey}.pdf                          # named after bib key (e.g., avis2018government.pdf)
  paywalled_urls.json                    # URLs for manual download on campus

The naming chain: literature.md → file: avis2018government.pdf → bib key @article{avis2018government, ...} → PDF avis2018government.pdf. One identifier across all three locations.

Update mode (--update <citekey>)

Surgical single-entry edit. Use when one citation in literature.md needs a note refreshed after a /next iteration — typically because the run produced a result that connects to or contradicts the paper.

Minimal read set:

1. $PROJ/CLAUDE.md — current focus.
2. $PROJ/docs/literature.md — locate the entry under its topic heading.
3. $PROJ/paper/*.bib — verify the citekey exists and the metadata is consistent.
4. The --artifact build path (if given) — the result that motivated the update.

Do not re-run searches, re-walk citation graphs, re-curate the candidate list, or re-fetch PDFs. The update touches one annotation, not the bibliography pipeline.

What to edit: only the target citekey's entry — typically its relevance note ("Tests prediction X on data Y", "Superseded by Z 2024"), not its bibliographic metadata. If metadata is wrong, fix it in paper/*.bib and note in the report.

What not to touch: other entries, the topic heading structure, the citekey convention.

Output: edited literature.md plus a one-paragraph summary of what changed. The summary belongs in the /next end-of-iteration report.

Audit mode (--audit / audit-bib)

The discovery pipeline's hallucination guard (step 4, Gotchas) is preventive: never add a paper the APIs didn't return. Audit mode is the detective complement — it checks a bib that has already accreted entries from many sources (coauthors, hand edits, Claude drafting the paper) and flags ones whose metadata doesn't match the authoritative record. Run it whenever entries were added outside this pipeline, and before submitting a paper.

# Audit only the entries actually cited in the paper (the printed reference list):
python3 $ROOT/research-kit/tools/literature_search.py audit-bib \
    --bib $PROJECT/paper/references.bib \
    --cited-from $PROJECT/paper/*.tex

# Audit every entry in the bib:
python3 $ROOT/research-kit/tools/literature_search.py audit-bib \
    --bib $PROJECT/paper/references.bib

# Machine-readable:  add --json

What it does, per entry (all verdicts come from a live Crossref lookup):

• has a DOI → fetch Crossref; compare title, first-author surname, and year.
  • TITLE-MISMATCH (HIGH) — the DOI resolves to a different paper (wrong/ copy-pasted DOI, or a fabricated title on a real DOI).
  • AUTHOR-MISMATCH (MED) — title matches but the lead author doesn't.
  • YEAR-OFF (LOW) — year matches neither the online nor the print year (online-first vs issue-year ambiguity is accepted, not flagged).
  • DOI-UNRESOLVED (HIGH) — the DOI doesn't exist at Crossref.
• no DOI → Crossref title search.
  • DOI-FOUND (INFO) — a matching DOI exists; consider adding it.
  • SUSPECT (HIGH) — an @article/@inproceedings with no DOI and no Crossref title match: a likely hallucination, verify by hand.
  • NO-DOI-MANUAL (INFO) — @misc/@book/news/data source that can't be auto-verified; check the URL/source yourself.
• MISSING-ENTRY (HIGH) — a key cited in the .tex has no bib entry.

Exits non-zero if any HIGH finding, so it can gate a pre-submission check.

Cardinal rule: the tool reports, it never rewrites the bib — and it never "fixes" an entry from model memory. When acting on a finding, look the correct value up (Crossref/SSRN/publisher) and verify before editing. Guessing a replacement DOI from memory is how the hallucination got there; a wrong DOI that happens to resolve (e.g. 10.1093/pan/mpr014 → a different fraud paper) will pass a careless "fix" and fail this audit.

Gotchas

• Rate limits: Semantic Scholar allows ~100 req/sec unauthenticated but can throttle. The script has built-in retry with backoff. If many queries fail, wait a few minutes and re-run with --refresh.
• Hallucination prevention: Never add a paper to literature.md that is not in candidates.json. All metadata must come from APIs, not from Claude's memory. If a paper seems relevant but wasn't found by the APIs, add it to queries and re-search rather than inventing the entry.
• Existing literature.md: When updating, merge with existing entries. Do not remove entries that were previously curated — only add new ones.
• Bib entry quality: Crossref metadata is authoritative. If crossref-verify fails for a DOI, flag the entry with a % UNVERIFIED comment in the bib file.
• PDF downloads: Some OA URLs point to HTML landing pages, not PDFs. The script checks Content-Type and skips non-PDFs. These will appear in the paywalled list.
• Paywalled PDFs cannot be downloaded programmatically. Python requests, curl, headless Chromium (Playwright/Selenium), and even headless Chrome with copied user profiles all get blocked by publisher bot detection (403 errors). The only method that works is the user's actual Chrome browser with the AlwaysOpenPdfExternally policy set. Do not waste time trying programmatic approaches — go straight to the HTML download page method.
• Chrome Preferences vs policies: Editing ~/.config/google-chrome/Default/Preferences while Chrome is running has no effect — Chrome holds prefs in memory and overwrites the file on exit. Use /etc/opt/chrome/policies/managed/*.json instead, which takes effect after clicking "Reload policies" on chrome://policy. Policies require sudo to write.
• Chrome PDF viewer vs download: By default Chrome renders PDFs inline in iframes and tabs. The AlwaysOpenPdfExternally policy only applies to top-level navigation, not iframes. The download page must use <a> clicks with target="_blank", not iframes or fetch().
• Publisher filename renaming: Publishers save PDFs with their own names (e.g., w31266.pdf, rest_a_01101.pdf, Elsevier PII codes). After each download round, check for new files and rename to citekey format. Some downloaded files may be wrong (appendices, unrelated papers from mismatched DOIs) — verify file size and spot-check with pdftotext.
• Check existing PDFs first: ~/Dropbox/literature/ contains ~4000 PDFs from prior work. Always search there before downloading. Filenames follow the pattern Author - Year - Title.pdf.
• Dropbox folder naming: Project Dropbox folders may not match the git slug exactly. Always ls ~/Dropbox/ and find the right folder before writing.
• Git tracking: docs/literature/ (candidates.json, queries, search log) should be committed to git. PDFs should NOT be in git — they go to Dropbox only.
• Clean up Chrome policy after downloading — remove /etc/opt/chrome/policies/managed/download_pdf.json so Chrome returns to normal PDF viewing behavior.