BioProject Metadata Retrieval

Overview

This skill retrieves NCBI BioProject records through the official E-utilities API and exports them as CSV/JSON.

The alternative — scraping the BioProject web pages — is fragile in a way that is easy to underestimate. Page-scraping code ends up with selectors like div:nth-child(5) that encode the current number of wrapper elements in the page. When NCBI adjusts its layout, the selector silently stops matching and the description column comes back empty, which is indistinguishable from "this project has no description".

E-utilities returns structured XML with stable field names. The same query that works today keeps working after a website redesign.

The skill also follows NCBI's usage policy properly: a contact email is required (via --email or the BIOPIPE_EMAIL environment variable). NCBI uses the email to identify callers and to contact them when problems occur; fabricating placeholder addresses defeats that purpose and can get your IP rate-limited.

When to Use This Skill

• Finding out what a specific BioProject / PRJNA / PRJEB accession contains
• Collecting titles, descriptions and organisms for a list of projects
• Screening projects by keyword before deciding which ones to download
• Building a project catalogue for a methods or dataset section
• Looking up submitter and release dates for citation purposes

Quick Start

export PYTHONPATH="$PWD/packages"
export BIOPIPE_EMAIL="you@your-institution.edu"

# 从文件读取 accession
python scripts/search_bioprojects.py --ids prj_ids.txt --output-dir results/

# 直接给 accession
python scripts/search_bioprojects.py --ids PRJNA172563 PRJNA212358 --output-dir results/

# 只保留描述里提到 CRISPR 的项目
python scripts/search_bioprojects.py --ids prj_ids.txt --output-dir results/ --filter CRISPR

# 有 NCBI API key 时可提高限速（10 req/s）
python scripts/search_bioprojects.py --ids prj_ids.txt --output-dir results/ --api-key $NCBI_API_KEY

Workflow

1. Collect accessions — --ids accepts accession strings or paths to text files (one per line, # for comments). Entries that do not look like BioProject accessions are reported and skipped rather than silently dropped.
2. Resolve to UIDs — esearch translates accessions into NCBI's internal UIDs. The query uses the [Project Accession] field qualifier, without which an accession string can match unrelated records.
3. Fetch details from two endpoints — esummary and efetch are called together and their results merged. This is not redundancy: no single endpoint carries every field. esummary is the only source for the structured organism name, taxonomy ID, project type and submitter organisation; efetch is the only source for the release date and the full description text. Calling only one of them silently drops about half the metadata without raising an error.
4. Parse — the merged records carry accession, title, name, description, organism, taxonomy ID, project type, submitter, registration date, release date and last-update date. All three NCBI date formats are normalised to YYYY-MM-DD.
5. Filter (optional) — --filter keeps records whose title, name or description contains the keyword (case-insensitive). --drop-noisy removes descriptions that look like page navigation rather than content.
6. Export — bioproject_data.csv and bioproject_data.json are written to the output directory.

Parameters

Outputs

Validation

• A description should be a paragraph, not a sentence fragment. Empty or one-word descriptions usually mean the project record genuinely lacks one — some submitters provide only a title. Cross-check with the linked BioSample or SRA records.
• The accession prefix indicates the archive — PRJNA (NCBI), PRJEB (EBI/ENA), PRJDB (DDBJ). A project submitted to ENA may be reachable through ENA's own portal rather than NCBI's.
• Organism names come from the submitter and are not normalised. A project may list several organisms, in which case the field contains a comma-separated list rather than a single taxon — do not treat it as a single taxid. The organism field always holds a name; the numeric identifier belongs in taxid.
• An empty date means "not recorded", not "zero". NCBI uses the placeholder 1/01/01 00:00 for "date unknown"; it is deliberately not converted into a real date, so the field stays empty. Do not interpret an empty registration_date as a bug — check whether the project is dbGaP-backed (those records have their submitter details stripped).
• Release date may be in the future or absent for embargoed projects. An embargoed project returns its metadata to authorised submitters only; the public record will be sparse.
• Search term semantics — [Project Accession] matches the accession field exactly. To search descriptions by keyword across all projects rather than a known list, use NCBI Entrez directly with bioproject[Filter] and a keyword term.
• Search term semantics — [Project Accession] matches the accession field exactly. To search descriptions by keyword across all projects rather than a known list, use NCBI Entrez directly with bioproject[Filter] and a keyword term.

References

• references/methodology.md — E-utilities protocol, field semantics, rate limits
• references/troubleshooting.md — common failures and how to diagnose them
• NCBI E-utilities documentation: https://www.ncbi.nlm.nih.gov/books/NBK25501/
• BioProject: https://www.ncbi.nlm.nih.gov/bioproject/