polars-dovmed

Search the PubMed Central Open Access and bioRxiv corpora with polars-dovmed, through the hosted API or local parquet files.

SKILL_DIR is the directory that contains this SKILL.md, installed at ~/.agents/skills/polars-dovmed. Run the helper as uv run --script "$SKILL_DIR/scripts/query_literature.py" so its pinned dependency is available. The helper loads ~/.config/polars-dovmed/.env.

Access

• Hosted API: needs POLARS_DOVMED_API_KEY in the environment or in the .env file. Never pass the key in argv, and never write it to run directories, memory records, summaries, or answers. The API root (https://api.newlineages.com/) states that keys are issued by the admin only.
• Local corpora: upstream dovmed download, dovmed build-parquet, and dovmed scan from https://github.com/UriNeri/polars-dovmed. omics-skills ships neither an API key nor the corpora.
• With neither, state that polars-dovmed is not configured and use another literature skill.

Instructions

1. Create a run directory, tasks/polars-dovmed-runs/<date-topic>/, outside $SKILL_DIR. Save prompt.txt, query.json, payloads, raw responses, timing files, and any curated summary there. Do not commit generated runs.
2. Write query.json (format below) and inspect it before searching: spelling, taxonomy aliases, regex escaping, and noisy terms.
   • Start anchor-only: exact entity names and evidence-based aliases. For prompts such as "hosts of X" or "distribution and genomics of X", search X and its aliases first and use the topic words for triage or a second pass. Never add generic words (distribution, genome, environment, host, taxonomy) as peer OR concepts.
   • Do not invent aliases by splitting compact taxon, gene, or clade names. Keep a real multi-word synonym as one phrase term.
   • Put acronym collisions and wrong systems in disqualifying_terms.
3. Run discovery first, one corpus per call. Never start with --corpus both.
   • bioRxiv: async discovery with --poll-timeout 75 and --details-rerank-limit 8 to 12.
   • OpenPMC: --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank --poll-timeout 120. Map a requested year range to its bands and search only those (table below). Never run a broad unbanded OpenPMC scan. Fetch details by PMCID after inspecting the top hits.
   • Use --extract-matches none --add-group-counts primary for discovery.
   • Wrap calls in timeout when it exists.
4. Inspect the first 5 to 10 hits. If they are noisy, refine query.json and rerun.
5. Fill citation metadata from the corpus details first, then from --crossref-metadata or /crossref-lookup. Use DOI landing pages only when corpus metadata and Crossref disagree or stay ambiguous; do not use generic web search for DOI repair.
6. When speed matters, record wall time with shell time -p or timeout, plus the helper's elapsed_ms. Timeouts and failed endpoints are results; report them.
7. Do not use the flat /api/search_literature endpoint. It needs --allow-flat-query and can hang behind the edge proxy.

Query JSON

{
  "anchor_entity": [["primary_name"], ["alias_1"], ["alias_2"]],
  "relation_or_property": [["primary_name", "relation_term"]],
  "disqualifying_terms": [["term_to_exclude"]]
}

Both search paths (FTS and parquet scan) apply the same logic: terms in one inner group are AND'd and must co-occur; inner groups in a concept are OR'd; concepts are OR'd, and items matching more groups rank higher. A multi-word term matches as an exact phrase. List each synonym as its own single-term group; use a multi-term group only to require co-occurrence.

Rank hits in this order: anchor in title, anchor in abstract, anchor plus support term in title or abstract, several distinct group matches, full-text-only matches. Down-rank hits that match only generic support terms.

Year bands (OpenPMC)

For a range that crosses bands, pass a comma list, for example --year-bands 2021_2023,2024_plus.

Hosted API commands

RUN=tasks/polars-dovmed-runs/mirusviricota-hosts-$(date +%Y%m%d)
mkdir -p "$RUN"
printf '%s\n' "papers describing hosts of Mirusviricota" > "$RUN/prompt.txt"
# Write and inspect "$RUN/query.json" before searching.

# bioRxiv discovery
timeout 90s uv run --script "$SKILL_DIR/scripts/query_literature.py" \
  --queries-file "$RUN/query.json" --corpus biorxiv --mode discovery \
  --extract-matches none --add-group-counts primary \
  --max-results 25 --details-rerank-limit 12 --poll-timeout 75 \
  --save-payload "$RUN/payload_biorxiv.json" --save-response "$RUN/results_biorxiv.json" \
  > "$RUN/summary_biorxiv.json" 2> "$RUN/time_biorxiv.txt"

# OpenPMC discovery over the four clean year bands
timeout 120s uv run --script "$SKILL_DIR/scripts/query_literature.py" \
  --queries-file "$RUN/query.json" --corpus pmc --mode discovery \
  --sync --year-bands clean_split --year-band-workers 4 --skip-details-rerank \
  --extract-matches none --add-group-counts primary \
  --max-results 25 --poll-timeout 120 \
  --save-payload "$RUN/payload_pmc.json" --save-response "$RUN/results_pmc.json" \
  > "$RUN/summary_pmc.json" 2> "$RUN/time_pmc.txt"

# Details for known PMCIDs (bioRxiv: pass DOIs with --corpus biorxiv),
# with bounded Crossref fill-in for missing DOI, year, or journal
uv run --script "$SKILL_DIR/scripts/query_literature.py" \
  --details PMC6362216 PMC10132079 --corpus pmc \
  --crossref-metadata --crossref-limit 10 \
  --save-payload "$RUN/payload_details.json" --save-response "$RUN/results_details.json"

Local commands

uv run --script "$SKILL_DIR/scripts/query_literature.py" \
  --execution-mode local --corpus pmc \
  --local-parquet-pattern "$DOVMED_PMC_PARQUET" \
  --queries-file "$RUN/query.json" \
  --save-payload "$RUN/payload_local.json" --save-response "$RUN/results_local.json"

• --corpus pmc reads DOVMED_PMC_PARQUET and --corpus biorxiv reads DOVMED_BIORXIV_PARQUET unless --local-parquet-pattern is given. --corpus both needs one compatible explicit pattern; otherwise run separate scans.
• Local runs record --corpus-revision (or DOVMED_CORPUS_REVISION) and read flattened.csv, falling back to processed.parquet (or the legacy prcoessed.parquet) only when the compact output is absent. --timeout caps a local scan (default 900 seconds).

Reachability and smoke check

curl -sS --max-time 20 https://api.newlineages.com/ returns service metadata. It proves the service is reachable, not that a corpus scan works. The smoke check runs one bounded bioRxiv discovery and one details lookup:

uv run --script "$SKILL_DIR/scripts/smoke_test.py" --run-dir tasks/polars-dovmed-runs/smoke-test

Run either check before declaring an outage.

Input Requirements

• A literature question, turned into an inspected query.json.
• A hosted API key or local parquet files for the requested corpus.
• A writable run directory outside $SKILL_DIR.

Output

• prompt.txt, query.json, payload JSON, raw response JSON, timing files, and an optional curated summary in the run directory.
• A paper list with titles, identifiers, corpus, relevance notes, and timing.
• Warnings for timeouts, HTTP 502, Database not found, missing metadata, or fallback use.

Quality Gates

• The API key stayed in the environment and appears in no artifact.
• The run directory is outside $SKILL_DIR.
• query.json was written anchor-first and inspected before the search.
• Each corpus ran as its own discovery call; OpenPMC used clean year bands with --sync and --skip-details-rerank, limited to the requested years.
• The flat endpoint, --corpus both, and unbanded OpenPMC scans were avoided, or the reason is stated.
• Every hosted call had a poll timeout and, where available, a wall-clock timeout.
• The first 5 to 10 hits were reviewed and noisy terms refined.
• Citation metadata came from corpus details, then bounded Crossref, not generic web search.
• Timings and failures are reported when speed is part of the request.

Troubleshooting

Issue: --corpus both returns HTTP 502. Solution: Run --corpus biorxiv and --corpus pmc separately and merge the results.

Issue: OpenPMC year-band search returns Database not found. Solution: Report that the deployment does not expose the indexed OpenPMC bands. Do not fall back to an unbanded scan.

Issue: OpenPMC year-band search times out. Solution: Report the timeout and keep the bioRxiv result. Narrow to the requested bands or refine the anchors; do not retry with longer waits or an unbanded scan.

Issue: A "topic of X" query returns generic topic papers. Solution: Rerun with the entity and its aliases only, and use topic terms during triage.

Issue: Raw urllib requests get HTTP 403 with Cloudflare error 1010. Solution: Cloudflare rejects Python's default user agent. Use the helper, or send both X-API-Key and a User-Agent header.

Issue: /usr/bin/time is missing. Solution: Use shell time -p or record start and end timestamps.