ENA Sequencing Data Retrieval

Overview

This skill retrieves raw sequencing data (FASTQ) from the ENA Portal API and exports run-level metadata. It replaces a common but fragile pattern: scraping archive web pages to extract download links and file sizes.

The ENA Portal API returns structured fields — fastq_ftp, fastq_bytes, fastq_md5, library_strategy, read_count, base_count — for every run. That removes three failure modes at once:

• Fragile HTML parsing. Page-scraping approaches depend on absolute XPath positions (/html/body/div[2]/div[2]/div[2]/...) that break whenever the archive changes its layout. The API's schema is stable.
• Only getting .sra. The archive's own download links point at the original submitted format (usually .sra), which then needs a separate fastq-dump conversion step. ENA already ships the converted FASTQ, with per-file sizes and MD5 checksums.
• No integrity check. A truncated download looks like a successful one until a downstream tool fails on malformed reads. ENA publishes the MD5 of every FASTQ; this skill verifies each file after transfer and deletes it if the checksum does not match.

Downloads are resumable: an interrupted transfer continues from the byte it stopped at using an HTTP Range request, instead of starting over.

When to Use This Skill

• Fetching FASTQ for a list of SRA/ERR/DRR accessions for an RNA-seq project
• Collecting run metadata (library strategy, read counts, organism) before deciding what to download
• Resuming a large download that was interrupted
• Verifying that downloaded FASTQ files are complete and uncorrupted
• Sizing a dataset before committing bandwidth (metadata-only mode)

Quick Start

export PYTHONPATH="$PWD/packages"

# 批量下载
python scripts/download_runs.py --ids SRR390728 SRR1975568 --output-dir fastq/

# 从文件读取 accession 列表
python scripts/download_runs.py --ids runs.txt --output-dir fastq/

# 只查元数据不下载（先摸清数据规模）
python scripts/download_runs.py --ids runs.txt --metadata-only --output-dir meta/

# 只处理 RNA-Seq 文库，跳过 WGS 等其它类型
python scripts/download_runs.py --ids runs.txt --strategy RNA-Seq --output-dir fastq/

# 单端数据只取第一个 FASTQ
python scripts/download_runs.py --ids runs.txt --single-end --output-dir fastq/

Workflow

1. Collect accessions — --ids accepts accession strings directly or paths to text files (one accession per line, # for comments). Unlike the original script, there is no separate "index" configuration file mixing paths and URLs together; the accession list is the only input.
2. Query the API — ena_client.fetch_run_metadata() builds an run_accession=X OR run_accession=Y query and posts it to the ENA search endpoint. (The filereport endpoint rejects multi-accession queries, so search is used for batching.)
3. Export metadata — every run is written to ena_run_metadata.csv and .json, including library_strategy, read_count, base_count and the FASTQ paths, sizes and MD5s.
4. Filter (optional) — --strategy keeps only runs with a given library strategy, e.g. RNA-Seq.
5. Download — each FASTQ is fetched as a stream. Existing files are skipped unless --force is given; partial files are continued with a Range header.
6. Verify — the file's MD5 is compared against the ENA-published value. A mismatch deletes the file (so it is not mistaken for complete on the next run) and counts the run as failed.
7. Report — counts of downloaded / skipped / failed files, plus the list of failed runs.

Parameters

Outputs

Validation

• The FASTQ is valid gzip — gzip -t file.fastq.gz should pass. A file that fails this check was likely truncated; re-download with --force.
• MD5 matches — the metadata CSV records ENA's expected MD5 per file. The skill already verifies this, and deletes mismatching files.
• File size matches fastq_bytes — the metadata records ENA's byte count for each file. A size mismatch means the transfer was incomplete even if the MD5 was skipped with --no-verify.
• Read count is plausible — read_count in the metadata is the number of reads (not read pairs). For paired-end data the number of FASTQ files should be 2, and the total base count should be roughly read_count × read_length.
• Library strategy is what you expect — WGS runs also live in SRA/ENA. If you expected RNA-Seq and see WGS, the accession may belong to a different project; check --strategy filtering output.
• Spot-check the first reads — zcat file.fastq.gz | head -4 should show a header line starting with @ and a quality line starting with +.

References

• references/methodology.md — API semantics, format choice, integrity verification strategy
• references/troubleshooting.md — common failures and how to diagnose them
• ENA Portal API: https://www.ebi.ac.uk/ena/portal/api