Genomics workflow orchestration: engines, execution & acceleration

Last reviewed: 2026-07-03 — see the freshness policy.

Learning objectives

After this chapter you will be able to:

• Compare CWL, WDL, and Nextflow and choose one for a given team and portability need.
• Map each engine onto Slurm/HPC, cloud instances, and containers across AWS, GCP, and Azure.
• Choose between Illumina DRAGEN and NVIDIA Parabricks for acceleration, and know the trade-off each makes.

From "which store" to "how does the pipeline actually run"

Sequencing pipelines introduced the FASTQ→BAM→VCF flow and named Nextflow/nf-core as a workflow-engine option. This chapter is the deeper orchestration layer: which workflow language, on which execution substrate, on which cloud, with how much acceleration — the decisions that determine whether a genomics platform is portable, fast, or affordable (usually pick two).

The three workflow languages

flowchart TB
  Choice{"Workflow language"} --> NF & WDL & CWL
  NF["Nextflow<br/>language + engine combined"] --> NFnote["Most mature; the de-facto<br/>default in production genomics"]
  WDL["WDL<br/>language only, needs an engine<br/>(Cromwell, miniwdl, dxWDL)"] --> WDLnote["Strong in the Broad/Terra/clinical<br/>ecosystem; versioned spec"]
  CWL["CWL<br/>vendor-neutral standard,<br/>many independent implementations"] --> CWLnote["Best cross-platform portability;<br/>favored where multiple vendors<br/>must run the same pipeline"]

Rule of thumb: default to Nextflow for production velocity, reach for WDL if you're embedded in the Broad/Terra/GATK ecosystem, and choose CWL when true multi-vendor portability outranks convenience — e.g. a consortium pipeline that must run unmodified on several institutions' independent infrastructure.

Execution substrates

The same workflow definition can run on very different compute underneath it:

• Slurm / on-prem HPC — a traditional job scheduler across a fixed cluster. Cost-predictable once built, but capacity-bound; bursting past your cluster's size means queueing, not elastic scale. See on-premises & hybrid.
• Cloud instances/VMs — elastic compute (EC2, GCE, Azure VMs), typically orchestrated via the cloud's batch scheduler rather than raw instances directly.
• Containers (Docker, Singularity/Apptainer) — package the tool + dependencies once, run identically everywhere; nearly every modern genomics pipeline ships as containerized steps regardless of which workflow language or substrate executes them. Containers are what make the "write once, run on Slurm or cloud" promise of Nextflow/WDL/CWL actually true in practice — without them, portability is theoretical.

Cloud mapping

flowchart LR
  Engines["Nextflow / WDL / CWL"] --> AWS_["AWS: Batch (raw) or<br/>HealthOmics (managed workflows)"]
  Engines --> GCP_["GCP: Batch<br/>(successor to the retired<br/>Cloud Life Sciences API)"]
  Engines --> Azure_["Azure: Batch"]
  Engines --> DNAnexus_["DNAnexus<br/>(managed, multi-cloud:<br/>AWS + GCP + Azure)"]

Design implication: if cloud portability matters more than any single vendor's managed convenience, standardize on containerized Nextflow/WDL/CWL and treat AWS Batch/HealthOmics, GCP Batch, Azure Batch, or DNAnexus as interchangeable execution backends — this is the genomics-specific instance of the cloud portability principle: push the vendor-specific service to the edge, keep the workflow definition portable. When you need that portability across infrastructure you don't own — a multi-institution consortium or national genomics platform — GA4GH's Cloud work stream (WES, TES, TRS) adds a common API layer above these engines for exactly that case.

Acceleration: DRAGEN vs Parabricks

Both dramatically cut secondary-analysis time versus CPU-only pipelines, but they make different trade-offs:

Choose DRAGEN when you need its broader clinical functionality (pharmacogenomics, single-cell) and can absorb the license cost — common in accredited clinical labs already standardized on it. Choose Parabricks when standard WGS/WES secondary analysis at the lowest cost and GPU flexibility matters more than DRAGEN's extra functionality — see also NVIDIA for HLS for how Parabricks fits the broader NVIDIA healthcare stack.

Design guidance

1. Containerize every step, regardless of workflow language — this is what actually delivers "runs anywhere," not the language choice alone.
2. Decide your portability requirement before your engine. Multi-vendor consortium work pulls toward CWL; Broad/Terra-ecosystem work pulls toward WDL; general production velocity pulls toward Nextflow.
3. Model the DRAGEN license cost explicitly in any TCO comparison (see Trade-offs, TCO & cost) — it is usually the dominant line item, not the compute.
4. Treat GCP's Life Sciences API as gone. If you inherit an older pipeline referencing it, migrating to Batch is mandatory, not optional, as of mid-2025.

Check yourself

1. You need the same pipeline to run unmodified across three independent research institutions' infrastructure. Which workflow language fits best, and why?
2. Why is containerizing pipeline steps more important to portability than the choice of workflow language itself?
3. A clinical lab already licenses DRAGEN and needs pharmacogenomic star-allele calling. Why might Parabricks not be a drop-in replacement despite being cheaper?

Further reading

• nf-core pipelines · Cromwell (WDL engine) · Common Workflow Language
• DNAnexus · AWS HealthOmics · Google Cloud Batch
• NVIDIA Parabricks · Illumina DRAGEN