Core team tasks

This page provides detailed procedures for nf-core core team administrative tasks. Core team members rotate week ‘on-call’ duties to manage community operations and support pipeline development.

Review and onboard new pipelines

To approve and setup a new nf-core pipeline:

  1. Core: Update the project status on the issue to ‘Proposed’

  2. Community: Discuss benefits and limitations of the proposal:

    • Emphasise collaboration

      Note

      nf-core prefers to have more developers working on fewer pipelines, rather than many pipelines with a single developer.

    • Ensure proposal follows nf-core pipeline guidelines

    • In some cases, promote ‘partial acceptance’, i.e., if upstream steps already covered in another pipeline, propose new pipeline of just non-overlapping steps

  3. Core: Check proposal has received approval in nf-core/proposals from either:

    • Two members of the core team
    • One member of the core team and one member of the maintainers team

  4. Core: Make final decision:

    • If accepted: update the Project status on the issue to ‘Accepted’
    • If not accepted: update the project status to ‘Turned down’ (process ends here)

  5. Core: Verify that the pipeline name follows the pipeline naming guidelines

  6. Core: Create a slack channel with approved pipeline name:

  7. Core: Add the main developer(s) to:

  8. Core: Create a branch with the pipeline name in the nf-core/test-datasets repository

  9. Core/Dev: Set up the pipeline repository:

    • If development started in a custom (non-nf-core) repo, transfer the repository to nf-core
    • If starting fresh, create a new GitHub repository with the name of the pipeline

  10. Core: Add the pipeline name to the website YAML file designated to ignore this repository when building the website pipeline list

  11. Core/Dev: Use nf-core pipelines create to create a git repository and push it to GitHub

  12. Core/Dev: Once first release (mostly) ready:

    • Transfer the repository to nf-core GitHub organisation and remove the repository on the local machine/user account.

  13. Core/Dev: After first release, remove the pipeline name from the website YAML file ignoring this pipeline from the build, if it is listed

  14. Core/Dev: Trigger the GitHub action to sync the nf-co.re pipelines page:

    • Website repo > Actions > Build json files and md-cache > Run workflow > From main

Upload test data to a S3 bucket

To upload full-test AWS files for to a S3 bucket:

  1. Check the test data is valid.

    Note

    Small i nput data may be unreliable when downloaded from other repositories (e.g., limit total file size to < 10 GB total).

  2. Check that large files cannot be reduced or subset

  3. Select Amazon S3-managed keys (SSE-S3) and create the pipeline directory in the S3 bucket

Transfer a repository to nf-core

To transfer an existing repository to the nf-core GitHub organisation:

  1. Request repository ownership transfer from the developer
    • If the repository is under an organisation account, ask the developer to add a core team member as a co-owner
    • If the repository is under a personal user account, ask the developer to transfer ownership directly to a core team member’s personal account
  2. Transfer the repository to the nf-core organisation
    • In the repository, go to Settings, scroll to the bottom of the General page, and select Transfer
    • Confirm or update the repository name if needed
  3. Configure repository settings in the nf-core organisation

    • Ask Matthias for the repository ruleset JSON files

      Note

      These are still under development, but pinned in the core channel

    1. In the repository, go to Settings

    2. In the left sidebar, select Rules

    3. Select New ruleset

    4. Select Import a ruleset, then select a JSON file

    5. Select Create at the bottom of the page

      Warning

      Do not modify any fields

    6. Repeat the import and create steps for each JSON file provided

Create custom Docker containers for modules

To add a custom docker containers to the nf-core https://quay.io organisation:

Note

The person building the container must have organisation push rights. Only core members may have this access. New core members should request access from existing members.

  1. Check with the module author to ensure there is no available solution via Conda, Bioconda, or BioContainers

  2. Ask the module author to:

    • Place a Dockerfile alongside the module (sub)directories
    • Add a README.md describing why the container is needed, and provide instructions for building the container
    • (Optional) add a .gitignore file to ensure build artifacts are not pushed with the Dockerfile and README.md

  3. Ensure Docker is correctly authenticated for pushing to quay.io

    docker login --username <QUAY_USER_NAME> quay.io

  4. Build the Docker image locally and tag it appropriately:

    docker build . -t quay.io/nf-core/<TOOL>:<VERSION>

  5. Push the built container image:

    docker push quay.io/nf-core/<TOOL>:<VERSION>

  6. Share the container reference with the module author so they can update the module:

    container "nf-core/<TOOL>:<VERSION>"
    Note

    This replaces the entire Docker/Singularity condition in the module.

Activate Zenodo archiving of a new pipeline

To set up up Zenodo DOI:

Note

It’s recommended that a core team member transfers the DOI to the nf-core Zenodo community.

  1. Before release:

    1. Sign up and log in to Zenodo

    2. Select GitHub -> Connect Account in the dropdown menu of your account and connect your GitHub account

    3. Select GitHub in the dropdown menu of your account once connected

    4. Toggle the On switch of the pipeline to enable Zenodo archiving

      Note

      It’s a good idea to enable this for all active pipelines, as DOIs will only be assigned upon release.

      Note

      Some repositories (e.g., nf-core/exoseq) may not be activated as they are archived.

    5. Inform the pipeline developers to make a release

  2. Post release:

    1. Select Github in the dropdown menu of your account in Zenodo and find the relevant repository

    2. Click the Zenodo Record page for the release

    3. Find the Communities box on the record page and submit the record to the nf-core community

      Tip

      Google Chrome contains a graphical bug causing the Communities box not to show up, please use another browser if this happens to you.

    4. Copy the DOI for Cite all versions? in the Versions tab

    5. Update files on the pipeline master branch:

      • README.md: Add the Zenodo Badge and update the If you use this pipeline cite section
      • Nextflow.config: Update the manifest block to include the DOI
      • Commit these changes with the message “Add Zenodo ID after first release”

Add new community members to the GitHub organisation

To add a member to the nf-core GitHub organisation via the #github-invitations` channel

  1. Verify the request is reasonable (i.e., clearly not spam)
  2. Send an invitation via nf-core github teams
  3. Add new member to the nf-core contributors team
  4. Respond to the #github-invitations channel request by reacting with the ✅ emoji

Updating online bingo cards

To update the bingo cards to ensure they are relevant:

To update the bingo cards:

  1. Ask for the outreach google account credentials from @core-team
  2. Log in to Firebase Dashboard
  3. Select nfcore-bingo
  4. Select Realtime Database
  5. Select games and navigate down the tree until you reach lexicon
  6. Hover over lexicon and select + to add entries:
    • Key: Number
    • Value: New board entry
  7. Select the trash bin next to the given entry to remove it

Make backups

The following items should be regularly backed up by a member of core team with admin access to the relevant service:

  • HackMD: backup to Google Drive (Outreach)
  • Bingo cards rules JSON (realtime database): backup to Google Drive (Outreach)

Onboard a core team member

To onboard a new core team member:

  1. Add to core team Slack channel
  2. Add to GitHub core team
  3. Add as owners on the GitHub organisation
  4. Add to @core-team @outreach-team slack teams
  5. Add as admin on Slack
  6. Add to hackMD team as Admin
  7. Add to quay.io in team owners
  8. Add to Tower organisation as owner
  9. Add to nf-core google calendar(s)
  10. Add to core@nf-co.re forwarder
  11. Add to Zenodo community as admin
  12. Update Website / About (get new member to check)
  13. Ask new member to add nf-core apple core emoji as Status
  14. Make Slack announcement
  15. Make Social media announcement

Complete on-call duties

Duties for core team weekly ‘on-call’ rotation:

  • Facilitate discussions and handling acceptance/rejection procedures on nf-core/proposals (pipelines, RFCs, SIGs, etc.) => Github and Slack
  • Add new community members to the GitHub organisation => Github and Slack
  • Upload test data to AWS S3 buckets for new pipelines => Slack
  • Upload containers to quay.io
  • Distribute first release reviews => Slack
  • Activate Zenodo => Link
On this page