Nextflow / nf-core Pipeline Development

Configure, launch, and monitor nf-core bioinformatics pipelines for sequencing data analysis. Handles samplesheet creation, parameter tuning, resource configuration, and result interpretation.

Workflow

Step 1: Determine Pipeline and Input Data

Identify the appropriate pipeline based on the experimental design:

• RNA-seq (nf-core/rnaseq): Differential expression, transcript quantification, fusion detection.
• Sarek (nf-core/sarek): Germline and somatic variant calling from WGS/WES data.
• ATAC-seq (nf-core/atacseq): Chromatin accessibility, peak calling, differential accessibility.
• ChIP-seq (nf-core/chipseq): Histone modification and transcription factor binding.
• Methylseq (nf-core/methylseq): Bisulfite sequencing, methylation calling.

Validate input data:

• Check FASTQ file existence and read counts with zcat <file> | wc -l (divide by 4).
• Verify paired-end files have matching read counts.
• For public data, use nf-core/fetchngs to download from GEO/SRA accessions.

Step 2: Create the Samplesheet

Generate a valid samplesheet CSV matching the pipeline's expected format:

nf-core/rnaseq samplesheet:

sample,fastq_1,fastq_2,strandedness
CONTROL_REP1,/path/ctrl1_R1.fastq.gz,/path/ctrl1_R2.fastq.gz,auto
TREATMENT_REP1,/path/treat1_R1.fastq.gz,/path/treat1_R2.fastq.gz,auto

nf-core/sarek samplesheet:

patient,sample,lane,fastq_1,fastq_2
PATIENT1,TUMOR,L001,/path/tumor_R1.fastq.gz,/path/tumor_R2.fastq.gz
PATIENT1,NORMAL,L001,/path/normal_R1.fastq.gz,/path/normal_R2.fastq.gz

Validate the samplesheet against the pipeline schema before launching.

Step 3: Configure Pipeline Parameters

Set parameters based on the experiment:

nextflow run nf-core/<pipeline> \
  -profile docker \
  --input samplesheet.csv \
  --outdir results/ \
  --genome GRCh38 \
  -r <latest_stable_version> \
  -resume

Key parameter decisions:

• Aligner: For rnaseq, choose --aligner star_salmon (default) or --aligner hisat2. STAR+Salmon is preferred for most use cases.
• Variant caller: For sarek, configure --tools strelka,mutect2,snpeff based on germline vs somatic needs.
• Peak caller: For atacseq, MACS2 is default. Adjust --macs_gsize for non-model organisms.
• Resources: Use -profile to match compute environment: docker, singularity, conda, awsbatch, tower.

Step 4: Launch and Monitor

Execute the pipeline and monitor progress:

• Launch with nextflow run and capture the run name for tracking.
• Monitor with nextflow log <run_name> for task-level status.
• Check .nextflow.log for errors if tasks fail.
• Use -resume to restart from the last successful checkpoint after fixing issues.

Common failure resolutions:

• Memory errors: Increase memory in nextflow.config with process { withName: 'STAR_ALIGN' { memory = '64.GB' } }.
• Missing index: Ensure --genome matches an iGenomes reference or provide --fasta and --gtf for custom genomes.
• Container issues: Verify Docker/Singularity is running and the user has pull permissions.

Step 5: Interpret Results

Guide the user through key outputs:

• MultiQC report (multiqc_report.html): Overall run quality, adapter content, alignment rates, duplication levels.
• Pipeline-specific outputs: DESeq2 results (rnaseq), VCF files (sarek), peak files (atacseq).
• Quality thresholds: Flag samples with <70% alignment rate, >50% duplication, or <1M reads.

Output Format

## Pipeline Run Summary

**Pipeline**: nf-core/<name> v<version>
**Profile**: <compute_profile>
**Genome**: <reference>
**Samples**: <count>

### Launch Command
```bash
<full nextflow command>

Samplesheet

Key Parameters

Monitoring

• Run name: <name>
• Status: <running/completed/failed>
• Check progress: nextflow log <run_name>

Results Location

• MultiQC: results/multiqc/multiqc_report.html
• Counts: results/star_salmon/

## Examples

**User**: "Run RNA-seq analysis on these FASTQs with 3 control and 3 treated samples"
**Action**: Create samplesheet CSV with sample names and FASTQ paths, configure nf-core/rnaseq with STAR+Salmon aligner and GRCh38 genome, generate launch command.

**User**: "I have tumor and normal WES BAMs, I need variant calling"
**Action**: Configure nf-core/sarek with matched tumor-normal samplesheet, set `--step variant_calling --tools mutect2,strelka`, generate launch command.

**User**: "My ATAC-seq pipeline failed with a memory error on the MACS2 step"
**Action**: Check `.nextflow.log` for the failing task, add memory override in nextflow.config, relaunch with `-resume`.

## Best Practices

- Always use `-resume` to avoid recomputing completed steps after parameter changes or failures.
- Pin the pipeline version with `-r` (e.g., `-r 3.14.0`) for reproducibility. Never use `-latest` in production.
- Store the exact launch command and `nextflow.config` alongside results for provenance.
- For large datasets (>20 samples), use a cluster or cloud profile rather than local execution.
- Check the nf-core pipeline documentation for the specific version being used. Parameter names can change between major versions.
- When working with non-model organisms, provide `--fasta` and `--gtf` explicitly instead of relying on iGenomes.
- Use `nextflow clean` periodically to remove cached work directories and free disk space.
- For GxP environments, capture the Nextflow execution report (`-with-report`) and timeline (`-with-timeline`) for audit purposes.