Running nf-core pipelines offline

nf-core pipelines are designed to automatically fetch everything they need to run, including pipeline code, container images, and reference data. However, many high-performance computing (HPC) environments restrict internet access on compute nodes for security reasons. In such cases, you can still run nf-core pipelines by preparing all required components in advance on a system with internet access, then transferring them to your offline environment.

Offline setup

Running pipelines offline requires three main components:

• Nextflow runtime: The Nextflow workflow engine and its plugins
• Pipeline code: The nf-core pipeline itself, along with container images
• Reference data: Genome files and other reference datasets (if needed by your pipeline)

Transfer Nextflow offline

To transfer Nextflow to an offline system:

1. Install Nextflow in an online environment.

2. Run your pipeline locally.

   Note

   Nextflow fetches the required plugins. It does not need to run to completion.

3. Copy the Nextflow binary and $HOME/.nextflow folder to your offline environment.

4. In your Nextflow configuration file, specify each plugin (both name and version), including default plugins.

   Note

   This prevents Nextflow from trying to download newer versions of plugins.

5. Add the following environment variable in your ~/.bashrc file:

   export NXF_OFFLINE='true'

Transfer pipeline code offline

To transfer pipeline code to an offline system:

1. Run nf-core pipelines download <pipeline> in an online environment.

   Note

   This command will download the pipeline and configuration profiles, and packages everything into a .tar.gz file.

   Tip

   Add the argument --container singularity to fetch the singularity container(s).

2. Transfer the .tar.gz file to your offline system and unpack it.

   Note

   The archive contains directories called:

   • workflow: The pipeline files
   • config: nf-core/configs files
   • singularity: Singularity images (if you used --container singularity)
   Tip

   If you are downloading directly to the offline storage (e.g., a head node with internet access whilst compute nodes are offline), use the --singularity-cache-only option for nf-core pipelines download and set the $NXF_SINGULARITY_CACHEDIR environment variable. This reduces total disk space by downloading singularity images to the $NXF_SINGULARITY_CACHEDIR folder without copying them into the target downloaded pipeline folder.

Transfer reference genomes offline

To use nf-core reference genomes offline, download and transfer them to your offline cluster. See Reference genomes for more information.

Additional resources

For more information about running nf-core pipelines offline, see:

• nf-core/bytesize: Running pipelines offline