Testing

The key words “MUST”, “MUST NOT”, “SHOULD”, etc. are to be interpreted as described in RFC 2119.

1 Scope of testing

Tests for modules SHOULD be executable within the nf-core/modules GitHub repository CI with example test data.

Tests for modules MUST, at a minimum, run on the GitHub repository CI with a stub test that replicates the generation of (empty) output files and a versions file.

Module tests do not necessarily need to be able to execute ‘standalone’, i.e., run outside the nf-core/modules repository. For example, they don’t need to be executable within a pipeline repository.

2 Snapshots

Use only one snapshot per module test, which SHOULD contain all assertions present in this test. Having multiple snapshots per test will make the snapshot file less readable.

The snapshot SHOULD include all output channels for each test, or at a minimum, it MUST contain some verification that the file exists.

By default, the then block of a test should contain this:

assert snapshot(process.out).match()

When the snapshot is unstable, use another way to test the output files. See nf-test assertions for examples on how to do this.

3 Stub tests

A stub test MUST be included for the module.

4 Tags

Tags MUST be specified for any dependent modules to ensure changes to upstream modules will re-trigger tests for the current module.

tag "modules"
tag "modules_nfcore"
tag "<tool>"
tag "<tool>/<subtool>" // Only if there is a subtool
tag "<dependent_tool>/<dependent_subtool>" // Only if there is a tool this module depends on

5 assertAll()

Use the assertAll() function to specify an assertion, and there MUST be a minimum of one success assertion and versions in the snapshot.

6 Assert each type of input and output

Include a test and assertions for each type of input and output.

Use different assertion types if a straightforward process.out snapshot is not feasible.

7 Test names

Test names SHOULD describe the test dataset and configuration used. some examples below:

test("homo_sapiens - [fastq1, fastq2] - bam")
test("sarscov2 - [ cram, crai ] - fasta - fai")
test("Should search for zipped protein hits against a DIAMOND db and return a tab separated output file of hits")

8 Input data

Reference input data with the modules_testdata_base_path parameter:

file(params.modules_testdata_base_path + 'genomics/sarscov2/illumina/bam/test.paired_end.sorted.bam', checkIfExists: true)

9 Configuration of ext.args in tests

Module nf-tests SHOULD use a single nextflow.config to supply ext.args to a module. Define them in the when block of a test under the params scope.

config './nextflow.config'
 
when {
  params {
    module_args = '--extra_opt1 --extra_opt2'
  }
  process {
    """
    input[0] = [
      [ id:'test1', single_end:false ], // meta map
      file(params.modules_testdata_base_path + 'genomics/prokaryotes/bacteroides_fragilis/genome/genome.fna.gz', checkIfExists: true)
    ]
    """
  }
}

process {
  withName: 'MODULE' {
    ext.args = params.module_args
  }
}

No other settings should go into this file.

10 Skipping CI test profiles

If a module does not support a particular test profile, you can skip it by adding the path to corresponding section in .github/skip_nf_test.json.