Contributing a new pipeline

nf-core pipelines use a standardized template that provides file structure, boilerplate code, and git infrastructure for template synchronization. By starting with the nf-core template, you benefit from automated testing, consistent structure, and easier integration with the nf-core community.

Follow these steps to create a new nf-core pipeline, from initial proposal through to development. You will start by creating a new pipeline locally and working with it on your own GitHub account. Once you have a working version, you can request to move the repository to the nf-core GitHub organisation for final development, review, and release.

Request a new pipeline

To propose a new nf-core pipeline:

1. Create a proposal on the nf-core/proposals repository.

   • Use the dedicated issue template for new pipeline proposals.
   • Fill out the form with key information about the pipeline you want to write.

2. Wait for the proposal to be reviewed.

   • The proposal will be discussed and checked for uniqueness to ensure minimal overlap with existing pipelines.
   • Once accepted, the proposal will be added to the new pipeline tracking board on GitHub.
   • The core team will create a Slack channel for your pipeline.

3. Join the following Slack channels:

   • #pipeline-maintainers for major change announcements and general pipeline development discussion.
   • #release-review-trading to coordinate the two reviews required for your first release.

Create the pipeline from the template

To create a new pipeline from the nf-core template:

1. Create the pipeline using the nf-core command-line tool:

   nf-core pipelines create

   • See the nf-core tools documentation for detailed instructions.
   • This command creates the correct file structure, boilerplate code, and git infrastructure for template synchronization.

2. Switch to the dev branch:

   git checkout dev

   • All development should happen on dev or on other branches that get merged into dev.

Push to GitHub

To push your new pipeline to GitHub:

1. Create an empty repository on GitHub under your personal account.

   • Go to GitHub and select + then New Repository.
   • Do not initialise the repository with any files (README, LICENSE, or .gitignore). The nf-core template already includes these files.

2. Add the GitHub repository as a remote to your local git repository:

   git remote add origin https://github.com/<YOUR_USERNAME>/<YOUR_REPOSITORY>.git

3. Push all branches to the remote GitHub repository:

   git push --all origin

   • The nf-core pipelines create command generates three standard nf-core branches (master, dev, and TEMPLATE) with an initial commit shared between them.
   • This git structure is required for automatic template synchronisation.

You should now see the nf-core template and branches in the GitHub web interface.

Develop your pipeline

To develop your pipeline:

1. Work on the dev branch following standard git development practices.

2. Run the nf-core linting tool regularly to ensure your pipeline meets nf-core standards:

   nf-core pipelines lint

   • See the nf-core tools documentation for more information.
   • GitHub Actions also runs this automatically, and you will receive notifications if tests fail.

3. Test the pipeline locally with test data to verify:

   • The pipeline executes successfully.
   • Outputs are generated correctly.
   • No errors or warnings occur.

4. Use the debug profile when testing to get additional information:

   nextflow run <pipeline_name> -profile debug

   • This enables warnings about process selectors, shows additional debug output, and disables cleanup.

Set up test data

Test data is essential for validating your pipeline. nf-core uses the dedicated nf-core/test-datasets repository to store test data separately from pipeline code. This prevents repository bloat and ensures test data remains small and fast to execute.

Add test data to nf-core/test-datasets

To add your test data to the nf-core/test-datasets repository:

1. Fork the nf-core/test-datasets repository to your personal account.

2. Clone the repository and create a new branch for your pipeline:

   git clone https://github.com/<YOUR_USERNAME>/test-datasets.git
   cd test-datasets
   git checkout -b <pipeline_name>

3. Add your test data files.

   • Test data files must be very small due to GitHub file size limits and GitHub Actions timeout constraints.
   • Use minimal reference data such as PhiX, yeast, or part of a chromosome.
   • Aggressively subsample input data to be as small as possible while still testing pipeline functionality.

4. Ask for advice in the #test-data Slack channel before adding test data.

5. Push the new files to GitHub:

   git add .
   git commit -m "Add test data for <pipeline_name>"
   git push --set-upstream origin <pipeline_name>

6. Create a pull request against the main nf-core/test-datasets repository.

   • First, create a new branch with your pipeline name using the GitHub web interface on the nf-core/test-datasets repository.
   • Then open a pull request and select this branch as the target.

Configure the test profile

To configure the test profile in your pipeline:

1. Edit the conf/test.config file.

   • Replace the example URLs with URLs to your test data files in the nf-core/test-datasets repository.
   • View files on GitHub and select Raw to get the direct URL.

2. Add required parameters to ensure the pipeline runs with minimal additional flags.

   • The test profile can be combined with other profiles such as docker or conda.
   • Do not specify a hardware environment in the test configuration.

3. Run the pipeline with the test profile to verify it works:

   nextflow run <pipeline_name> -profile test,docker --outdir <OUTDIR>

4. Update the .github/workflows/ YAML files if you modify the test command.

Test data requirements

All nf-core pipelines must meet these test data requirements:

• The test profile must exist and work.
• The test should be as comprehensive as possible.
• The test should run as much of the pipeline as possible.
• The test dataset should be as small as possible while still testing pipeline functionality.

See the test data specifications for complete requirements.

Add full test data

You will need a full-size test dataset for the test_full profile:

1. Request upload to nf-core AWS S3 test buckets in the #request-core Slack channel.
   • This dataset produces realistic output when executed on each release.

Move your pipeline to nf-core

Once your pipeline is written and tests pass, you can request to add your pipeline to the nf-core organisation.

To move your pipeline to the nf-core organisation:

1. Announce on the #request-core Slack channel that you need a core team member to move your repository.

2. Transfer repository ownership to nf-core:

   • Go to your repository settings.
   • Under the General page, find the Danger Zone section.
   • Select Transfer Ownership and transfer to the nf-core organisation.

3. Fork the transferred repository back to your account to continue development.

Configure branches

To set up the branch structure:

1. Ensure your repository has dev and master branches.

   • The master branch contains the latest stable release code.
   • The dev branch contains the latest development code.

2. Set dev as the default branch before the first release.

   • This ensures users run the latest development code by default.
   • After the first release, set the default branch to master so users run the latest stable release by default.

Configure repository settings

Configure the following repository settings on GitHub:

• Add a description, the https://nf-co.re URL, and keywords.
• Enable Issues and disable Wiki and Projects.
• Protect the master branch to require review and passing tests.
• Set write permissions for nf-core/all and admin permissions for nf-core/admin.

Verify these settings using the nf-core repository health web page. This page reports the status of various checks and can fix errors automatically via the GitHub API.

Make the first release

Once your pipeline is written, tests pass, and the repository is in the nf-core organisation, you can prepare for your first release.

Reset the default branch

When the code is stable and ready for a release, set the default branch to master on GitHub.

Bump the version

To prepare for release:

1. Update the version number on the dev branch:

   nf-core pipelines bump-version --nextflow <new_version>

   • The first release should be version 1.0.0 (pre-releases are discouraged).
   • During development, use numeric versions with dev at the end (for example, 0.0.0dev).
   • Release versions must be numeric and follow Semantic Versioning.

Lint the pipeline

Run the linting tool with release checks to ensure your code meets nf-core standards:

nf-core pipelines lint --release

Request review

Your pipeline requires review by the nf-core community before release.

To get your pipeline reviewed:

1. Create a pull request from the dev branch to master on the nf-core repository.

   • This is a special case where all tests should pass.
   • Ensure all automated tests complete successfully.

2. Request two reviews, with at least one from the core or maintainers team.

   • Use the #release-review-trading Slack channel to coordinate reviews.

The review process depends on your master branch state:

• Clean master branch: If master contains only the initial commit, the pull request will represent all changes for review.
• Existing commits: The core team may merge your pull request and create a pseudo-pull request against the first commit to show all changes.

Reviewers will provide feedback on:

• README clarity and completeness
• Documentation quality, particularly output files and parameters
• Pipeline code structure and functionality

Complete the release

Once reviewers approve your pull request:

1. Add a changelog entry describing the pipeline functionality at release.

   • Describe the general features and capabilities.
   • Tag contributors with their GitHub handles so their icons appear on the release page.

2. Follow the pipeline release checklist to complete the release process.

The nf-core website and tools will automatically detect and display your new release.

Share your pipeline

Share your new pipeline with the community:

• Post in the #bytesize_suggestion Slack channel to arrange a 15-minute introductory talk about your pipeline.

Display your pipeline on the nf-core website

Verify that your pipeline appears on the nf-core website:

1. Check the list of ignored repositories.

2. If your pipeline appears in this list, create a pull request to remove it.

3. Request approval from a core team member.

Subsequent releases

After your first release, continue development following these practices:

1. Make pull requests against the dev branch on the nf-core repository.

   • All pull requests to dev should be reviewed before merging.

2. When ready for a new release:

   • Update the version number using nf-core pipelines bump-version.
   • Create a pull request against master.
   • If tests pass, the pull request can be merged and a new release made.

Additional considerations

Developing pipelines privately

While nf-core encourages open development, private development is possible with some considerations:

• Private pipelines receive the same review process as public pipelines.
• Follow the same development steps but use a private repository.
• Make the repository public before release, or request reviews in the #request-review Slack channel if you cannot make it public until after release.

If you plan to develop a pipeline privately and release through nf-core, contact the core team on Slack to discuss your strategy.