Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mtbseq workflow #8

Merged
merged 16 commits into from
Oct 30, 2024
Merged

Add mtbseq workflow #8

merged 16 commits into from
Oct 30, 2024

Conversation

Mxrcon
Copy link
Member

@Mxrcon Mxrcon commented Oct 28, 2024

MTBseq addition

Hey there 👋,

This PR adds

  • all required mtbseq modules
  • mtbseq workflows and sub workflows
  • parameters base configuration as used in mtbseq_nf
  • test profiles for MTBseq

the module structure follows:

── local
│   ├── mtbseq
│   │   ├── tbamend
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbbwa
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbfull
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbgroups
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbjoin
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tblist
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbpile
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbrefine
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbstats
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   ├── tbstrains
│   │   │   ├── environment.yml
│   │   │   ├── main.nf
│   │   │   └── meta.yml
│   │   └── tbvariants
│   │       ├── environment.yml
│   │       ├── main.nf
│   │       └── meta.yml
│   └── utils
│       └── rename_files.nf

I kindly request for your review @abhi18av and @ycode-sh for proceeding with this work.

PR checklist

  • Perform tests on the workflow
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

conf/mtbseq-base.config Outdated Show resolved Hide resolved
@Mxrcon Mxrcon marked this pull request as ready for review October 28, 2024 15:11
@Mxrcon Mxrcon self-assigned this Oct 28, 2024
Copy link

github-actions bot commented Oct 28, 2024

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit fcef247

+| ✅ 187 tests passed       |+
!| ❗  41 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
  • pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
  • pipeline_todos - TODO string in README.md: update the following command to include all required parameters for a minimal example
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in gatk4.nf: If in doubt look at other nf-core/modules to see how we are doing things! :)
  • pipeline_todos - TODO string in gatk4.nf: A module file SHOULD only define input and output files as command-line parameters.
  • pipeline_todos - TODO string in gatk4.nf: Software that can be piped together SHOULD be added to separate module files
  • pipeline_todos - TODO string in gatk4.nf: Optional inputs are not currently supported by Nextflow. However, using an empty
  • pipeline_todos - TODO string in gatk4.nf: List required Conda package(s).
  • pipeline_todos - TODO string in gatk4.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in gatk4.nf: Where applicable all sample-specific information e.g. "id", "single_end", "read_group"
  • pipeline_todos - TODO string in gatk4.nf: Where applicable please provide/convert compressed files as input/output
  • pipeline_todos - TODO string in gatk4.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in gatk4.nf: List additional required output channels/values here
  • pipeline_todos - TODO string in gatk4.nf: Where possible, a command MUST be provided to obtain the version number of the software e.g. 1.10
  • pipeline_todos - TODO string in gatk4.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "task.ext.args" directive
  • pipeline_todos - TODO string in gatk4.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
  • pipeline_todos - TODO string in gatk4.nf: Please replace the example samtools command below with your module's command
  • pipeline_todos - TODO string in gatk4.nf: Please indent the command appropriately (4 spaces!!) to help with readability ;)
  • pipeline_todos - TODO string in gatk4.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.2
  • Run at 2024-10-29 03:18:22

@abhi18av
Copy link
Member

@Mxrcon the major problem here is related to nf-validation plugin.

In the MTBseq-nf when we initiated the nf-schema plugin was fine but now it has been deprecated in favor of nf-validation.

plugin_includes - Found a nf-validation plugin import in ome/runner/work/tbanalyzer/tbanalyzer/workflows/mtbseqnf.nf, but nf-schema was used in nextflow.config

@abhi18av abhi18av self-requested a review October 28, 2024 15:42
@Mxrcon
Copy link
Member Author

Mxrcon commented Oct 28, 2024

@Mxrcon the major problem here is related to nf-validation plugin.

In the MTBseq-nf when we initiated the nf-schema plugin was fine but now it has been deprecated in favor of nf-validation.

plugin_includes - Found a nf-validation plugin import in ome/runner/work/tbanalyzer/tbanalyzer/workflows/mtbseqnf.nf, but nf-schema was used in nextflow.config

Hmm, just checked it, the port from nf-validation to nf-schema is very straightfoward, just added it and I'll now work into the test of the mtbseq workflow

@Mxrcon Mxrcon changed the title Mtbseq modules Add mtbseq workflow Oct 28, 2024
@Mxrcon
Copy link
Member Author

Mxrcon commented Oct 29, 2024

Update after Tests

Hey, I've performed local tests for the latest commits and here are the outputs:

Command executed:

nextflow run main.nf -profile conda -c mtbseq-test.conf --mode mtbseq -resume --mtbseq_parallel

executor >  local (27)
[89/24bbfd] process > NFCORE_TBANALYZER:MTBSEQ_NF:QUALITY_CHECK:RENAME_FILES (4781-04) [100%] 3 of 3 ✔
[66/e79bfd] process > NFCORE_TBANALYZER:MTBSEQ_NF:QUALITY_CHECK:FASTQC (4781-04)       [100%] 3 of 3 ✔
[c7/9408b0] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBBWA (4781-04)        [100%] 3 of 3 ✔
[3a/38d282] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBREFINE (4781-04)     [100%] 3 of 3 ✔
[67/526446] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBPILE (4781-04)       [100%] 3 of 3 ✔
[5f/05673e] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBLIST (4781-04)       [100%] 3 of 3 ✔
[49/2405d7] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBVARIANTS (4781-04)   [100%] 3 of 3 ✔
[8b/cb38bc] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBSTATS (cohort)       [100%] 1 of 1 ✔
[fc/6339ea] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBSTRAINS (cohort)     [100%] 1 of 1 ✔
[e1/c2aa8e] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBJOIN (cohort)        [100%] 1 of 1 ✔
[55/f73876] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBAMEND (cohort)       [100%] 1 of 1 ✔
[9d/39b91d] process > NFCORE_TBANALYZER:MTBSEQ_NF:PARALLEL_MODE:TBGROUPS (cohort)      [100%] 1 of 1 ✔
[00/a543e4] process > NFCORE_TBANALYZER:MTBSEQ_NF:MULTIQC                              [100%] 1 of 1 ✔
-[nf-core/tbanalyzer] Pipeline completed successfully-
Completed at: 28-Oct-2024 20:32:17
Duration    : 2h 16m 43s
CPU hours   : 6.6
Succeeded   : 27

Command Executed:

nextflow run main.nf -profile conda -c mtbseq-test.conf --mode mtbseq -resume

executor >  local (11)
[ea/1186c5] process > NFCORE_TBANALYZER:MTBSEQ_NF:QUALITY_CHECK:RENAME_FILES (3494-04) [100%] 3 of 3 ✔
[de/f3d790] process > NFCORE_TBANALYZER:MTBSEQ_NF:QUALITY_CHECK:FASTQC (3494-04)       [100%] 3 of 3 ✔
[24/281c56] process > NFCORE_TBANALYZER:MTBSEQ_NF:TBFULL (cohort)                      [100%] 1 of 1 ✔
[27/1965c8] process > NFCORE_TBANALYZER:MTBSEQ_NF:TBJOIN (cohort)                      [100%] 1 of 1 ✔
[6d/4c6c3b] process > NFCORE_TBANALYZER:MTBSEQ_NF:TBAMEND (cohort)                     [100%] 1 of 1 ✔
[de/d58589] process > NFCORE_TBANALYZER:MTBSEQ_NF:TBGROUPS (cohort)                    [100%] 1 of 1 ✔
[fe/859eb8] process > NFCORE_TBANALYZER:MTBSEQ_NF:MULTIQC                              [100%] 1 of 1 ✔
-[nf-core/tbanalyzer] Pipeline completed successfully-
Completed at: 28-Oct-2024 23:38:20
Duration    : 3h 5m 54s
CPU hours   : 3.0
Succeeded   : 11

Hope that this is enough for a merge, a I would like to proceed with pipeline documentation in another PR if possible.

@abhi18av, I would like to request a new review on this code before our next sync meet.

Kindly,
Davi

Copy link
Member

@abhi18av abhi18av left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Mxrcon , thank you for accommodating the change request and also for the end-to-end test for the MTBseq mode. I am going ahead with the merge and then we can focus on local MAGMA modules.

@abhi18av abhi18av merged commit 4767064 into dev Oct 30, 2024
3 of 6 checks passed
@abhi18av abhi18av deleted the mtbseq-modules branch October 30, 2024 09:48
@abhi18av
Copy link
Member

I have just tried to change the mode to method parameter in 4767064

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants