Adding ONT functionality to RAW and CLEAN #133

simonleandergrimm · 2024-12-18T20:42:22Z

Changes

Added Oxford Nanopore (ONT) data processing support:
- Added FILTLONG module for ONT read filtering and quality control
- Modified CLEAN workflow to use FILTLONG for ONT data and FASTP_SINGLE/PAIRED for Illumina data
- Added ONT detection in read_type.config based on samplesheet content
- Added ONT test data and configuration
Enhanced sequence length reporting:
- Added length statistics to MultiQC summarization
- Added length distribution plots to QC output
- Renamed and generalized MultiQC processing modules (removed "pair" suffix)
- Added new length statistics TSV output to results
Added staphb/filtlong:0.2.1 container to support ONT processing

Possible Todos (I don't think these should block merging)

Drop creation and publication of adapters -> not applicable for ONT atm.

…y/mgs-workflow into simon-ont-raw-clean

…in the name.

simonleandergrimm · 2024-12-18T21:36:14Z

@harmonbhasin Can you take an initial look and let me know if you see obvious flaws?

Here is what the output of the MultiQC output for ONT data looks like: https://data.securebio.org/simons-notebook/posts/2024-12-08-mgs-ont-raw-eval/

harmonbhasin

This all looks correct to me. The only thing I ask is that you confirm that the output for the length stats file looks good for single-end/paired-end short reads.

modules/local/summarizeMultiqc/main.nf

harmonbhasin

Just noticed that you added the ONT test to the nf-test file, but not to the yaml file that makes it run on Github Actions, please do this, thank you!

simonleandergrimm · 2025-01-06T18:56:14Z

Just noticed that you added the ONT test to the nf-test file, but not to the yaml file that makes it run on Github Actions, please do this, thank you!

Fixed

simonleandergrimm · 2025-01-06T19:47:29Z

@harmonbhasin This is the error I get when nf-test is run on github. nf-test works locally. Can you let me know if you see what might be wrong here?

Otherwise I adopted your suggestions. Once the nf-test thing is resolved this is good for review by Will.

Run nf-test test --tag run --verbose

🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr


End-to-end test of MGS workflow

  Test [4ae1c173] 'Test run workflow' 
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/home/runner/work/mgs-workflow/mgs-workflow/main.nf` [admiring_stallman] DSL2 - revision: 73db19e0f9
    > Downloading plugin [email protected]
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > Downloading plugin [email protected]
    > [94/67ea50] Submitted process > RUN:RAW:QC:FASTQC_LABELED (1)
    > [d0/3647f3] Submitted process > RUN:CLEAN:FILTER_READS (1)
    > [9f/eb6025] Submitted process > RUN:RAW:QC:MULTIQC_LABELED (1)
    > [24/d22e6[8](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:9)] Submitted process > RUN:CLEAN:QC:FASTQC_LABELED (1)
    > [5d/1c48d5] Submitted process > RUN:PROFILE:SUBSET_READS_TARGET (1)
    > [54/1f6453] Submitted process > RUN:CLEAN:QC:MULTIQC_LABELED (1)
    > Staging foreign file: s3://nao-testing/index-test/output/results/ribo-ref-concat.fasta.gz
    > [87/1fc755] Submitted process > RUN:EXTRACT_VIRAL_READS:BBDUK_HITS (1)
    > [6e/d760af] Submitted process > RUN:PROFILE:BBDUK (1)
    > [64/fdca31] Submitted process > RUN:CLEAN:QC:SUMMARIZE_MULTIQC (1)
    > [cf/8dea48] Submitted process > RUN:RAW:QC:SUMMARIZE_MULTIQC (1)
    > [76/f[9](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:10)1a7d] Submitted process > RUN:EXTRACT_RAW_READS_FROM_PROCESSED (1)
    > [40/942af5] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:BBMERGE (1)
    > [19/fa812b] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:BBMERGE (1)
    > [6f/0589c1] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:SUMMARIZE_BBMERGE (1)
    > [5c/b45ec3] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:JOIN_FASTQ (1)
    > [ae/b4ea7a] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:SUMMARIZE_BBMERGE (1)
    > [ad/f49455] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:JOIN_FASTQ (1)
    > [61/38d663] Submitted process > RUN:EXTRACT_VIRAL_READS:CUTADAPT (1)
    > Staging foreign file: s3://nao-testing/index-test/output/results/kraken_db
    > [9c/0e0866] Submitted process > RUN:RAW:QC:MERGE_MULTIQC_BASIC
    > [b3/6a0269] Submitted process > RUN:CLEAN:QC:MERGE_MULTIQC_QBASE
    > ERROR ~ Error executing process > 'RUN:RAW:QC:MERGE_MULTIQC_BASIC'
    > 
    > Caused by:
    >   Process `RUN:RAW:QC:MERGE_MULTIQC_BASIC` terminated with an error exit status (125)
    > 
    > 
    > Command executed:
    > 
    >   #!/usr/bin/env Rscript
    >   library(tidyverse)
    >   in_paths <- str_split("raw_concat_gold_standard_qc_basic_stats.tsv.gz", " ")[[1]]
    >   print(in_paths)
    >   tabs <- lapply(in_paths, function(t) read_tsv(t, col_names = TRUE, cols(.default="c")))
    >   for (t in tabs) print(dim(t))
    >   tab_out <- bind_rows(tabs)
    >   print(dim(tab_out))
    >   sapply(tabs, nrow) %>% sum %>% print
    >   write_tsv(tab_out, "raw_concat_qc_basic_stats.tsv.gz")
    > 
    > Command exit status:
    >   125
    > 
    > Command output:
    >   (empty)
    > 
    > Command error:
    >   c0e4127c6ef4: Retrying in 1 second
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   c0e4127c6ef4: Downloading
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Downloading
    >   802254caa4ee: Retrying in 15 seconds
    >   802254caa4ee: Retrying in 14 seconds
    >   802254caa4ee: Retrying in 13 seconds
    >   802254caa4ee: Retrying in 12 seconds
    >   802254caa4ee: Retrying in 11 seconds
    >   802254caa4ee: Retrying in [10](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:11) seconds
    >   802254caa4ee: Retrying in 9 seconds
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Retrying in 20 seconds
    >   802254caa4ee: Retrying in 19 seconds
    >   802254caa4ee: Retrying in 18 seconds
    >   802254caa4ee: Retrying in 17 seconds
    >   802254caa4ee: Retrying in 16 seconds
    >   802254caa4ee: Retrying in 15 seconds
    >   802254caa4ee: Retrying in 14 seconds
    >   802254caa4ee: Retrying in 13 seconds
    >   802254caa4ee: Retrying in 12 seconds
    >   802254caa4ee: Retrying in [11](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:12) seconds
    >   802254caa4ee: Retrying in 10 seconds
    >   802254caa4ee: Retrying in 9 seconds
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Downloading
    >   docker: unknown: <?xml version="1.0" encoding="UTF-8"?>
    >   <Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJHMEUCIQCeIV4tfQcJ4orTMrZ5u4B8veMnlidVp99W1l6joTQRugIgDanE0IlIJAZ/aX61tAmHihmGywEMl7B15456kFL7MtIq7gQISxAAGgw3MTAwMTUwNDA4OTIiDGBDvMDxo9r+sdUzOCrLBIlSzKWaOEko/eQmED0w2OtilOeboB1PHAMnTlzh+Ru6CJ/sLjjwyQ5XsSNbDbWPezZZ/oKk2D8XEERqY+Xgma0H+PbLFkqROMKnNFtv7zvXwI7uYZdsov7YvSHnRZ+8KIztxvam+iBia3QAPgpqoP1Cv9mNw0kkB8aSMvjBNmqMyDkp7VJI+EJtFLnxTKQPDMeDR9eT4mFynBXuSMvoHNqoybqG7K2IRQA5wnhr/P6eZZ5xF3OySxo3P5Co9VeOkNOLpx2WaeZ+TqksZdFTjQML+j7MRi5NLDI7HUWP/P47APIbXrVaj4WFIs2BKvVXUeFwIJ+GxSsRu66YoXie53jnDWZiRAvfUOA2P3AKGYoVXlmwxN3UBDrbgFtUI6twAzH2UTBfviNoYYXHDa2EFjIQ8bLM1LdN554CCncJRzIheCBVDARw8+LIeT4zTJCRPJ4cdSL2QOFvYQiN8m3nEn09rMAUXOMRfGnvMXVkHC4EUtDtZUrZxKdHvA9MzqtVKMfYV3NPJtqxF9nxBqu4GxUIVhKTBozZtTMJtfDV2QsuwgvT3ZpY0Vr3qtBOnxGZ/cqwqGnCEjiYcIg43wSxmrUDR0/9StBCR4Fb1QTUqGcBVScT2R5oE5CKLoFxXkcCx8FNYz5B8Gqj0JCfFuKc/gUZQdCBMDPTFO8cp5brNfqimGtwfzaleQ65Co/4/HdlUGLWT4xdcEPPzxFSc/XCy0C7XefOhCPyc9Tu+FeoJZ6GTaI5YxcdtQHn+282ScMZA61LyiQ6NIMX2zR3MJyz8LsGOpoBWqE7+eLUXpfwp/Im0R2QzUeZdR85rxggo2vfOIG+jpJtye5KUxhxKvKN5pIiFOnnVADVhQNskCdF1SyntDNBK4IqE41480C/LdFyPuI362qp3EIVZ/aamajzroZUfw8EDyx7c+PEn6iz2CbJNKKpfFcvxMKM4yxVhFUWZdr/Xig10rH8E/KCLFsvAEO4jljVwNgxJZpLq51fqw==</Token-0><RequestId>XXTBWVERT9Z1XXV5</RequestId><HostId>asY4icJxCgIPWPkzZkKvjdKhjE0Ukw4Hmr1mihJ7J7ah+7z8a8mg3h0sabPaUrepodnjw471hbM=</HostId></Error>.
    >   See 'docker run --help'.
    > 
    > Work dir:
    >   /home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/work/9c/0e0866d01d98ff46588f1ba7004635
    > 
    > Container:
    >   wave.seqera.io/wt/5ac75ac6191e/rocker/tidyverse:4.4.1
    > 
    > Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
    > 
    >  -- Check '/home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/meta/nextflow.log' file for details
    > Execution cancelled -- Finishing pending tasks before exit
    FAILED (390.383s)

  Assertion failed: 
  
  assert workflow.success
         |        |
         workflow false
  
  Nextflow stdout:
  
  N E X T F L O W  ~  version 24.10.3
  Launching `/home/runner/work/mgs-workflow/mgs-workflow/main.nf` [admiring_stallman] DSL2 - revision: 73db19e0f9
  Downloading plugin [email protected]
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  Downloading plugin [email protected]
  [94/67ea50] Submitted process > RUN:RAW:QC:FASTQC_LABELED (1)
  [d0/3647f3] Submitted process > RUN:CLEAN:FILTER_READS (1)
  [9f/eb6025] Submitted process > RUN:RAW:QC:MULTIQC_LABELED (1)
  [24/d22e68] Submitted process > RUN:CLEAN:QC:FASTQC_LABELED (1)
  [5d/1c48d5] Submitted process > RUN:PROFILE:SUBSET_READS_TARGET (1)
  [54/1f6453] Submitted process > RUN:CLEAN:QC:MULTIQC_LABELED (1)
  Staging foreign file: s3://nao-testing/index-test/output/results/ribo-ref-concat.fasta.gz
  [87/1fc755] Submitted process > RUN:EXTRACT_VIRAL_READS:BBDUK_HITS (1)
  [6e/d760af] Submitted process > RUN:PROFILE:BBDUK (1)
  [64/fdca31] Submitted process > RUN:CLEAN:QC:SUMMARIZE_MULTIQC (1)
  [cf/8dea48] Submitted process > RUN:RAW:QC:SUMMARIZE_MULTIQC (1)
  [76/f91a7d] Submitted process > RUN:EXTRACT_RAW_READS_FROM_PROCESSED (1)
  [40/942af5] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:BBMERGE (1)
  [19/fa8[12](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:13)b] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:BBMERGE (1)
  [6f/0589c1] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:SUMMARIZE_BBMERGE (1)
  [5c/b45ec3] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:JOIN_FASTQ (1)
  [ae/b4ea7a] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:SUMMARIZE_BBMERGE (1)
  [ad/f49455] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:JOIN_FASTQ (1)
  [61/38d663] Submitted process > RUN:EXTRACT_VIRAL_READS:CUTADAPT (1)
  Staging foreign file: s3://nao-testing/index-test/output/results/kraken_db
  [9c/0e0866] Submitted process > RUN:RAW:QC:MERGE_MULTIQC_BASIC
  [b3/6a0269] Submitted process > RUN:CLEAN:QC:MERGE_MULTIQC_QBASE
  ERROR ~ Error executing process > 'RUN:RAW:QC:MERGE_MULTIQC_BASIC'
  
  Caused by:
    Process `RUN:RAW:QC:MERGE_MULTIQC_BASIC` terminated with an error exit status (125)
  
  
  Command executed:
  
    #!/usr/bin/env Rscript
    library(tidyverse)
    in_paths <- str_split("raw_concat_gold_standard_qc_basic_stats.tsv.gz", " ")[[1]]
    print(in_paths)
    tabs <- lapply(in_paths, function(t) read_tsv(t, col_names = TRUE, cols(.default="c")))
    for (t in tabs) print(dim(t))
    tab_out <- bind_rows(tabs)
    print(dim(tab_out))
    sapply(tabs, nrow) %>% sum %>% print
    write_tsv(tab_out, "raw_concat_qc_basic_stats.tsv.gz")
  
  Command exit status:
    125
  
  Command output:
    (empty)
  
  Command error:
    c0e4127c6ef4: Retrying in 1 second
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    c0e4127c6ef4: Downloading
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Downloading
    802254caa4ee: Retrying in 15 seconds
    802254caa4ee: Retrying in 14 seconds
    802254caa4ee: Retrying in [13](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:14) seconds
    802254caa4ee: Retrying in 12 seconds
    802254caa4ee: Retrying in 11 seconds
    802254caa4ee: Retrying in 10 seconds
    802254caa4ee: Retrying in 9 seconds
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Retrying in 20 seconds
    802254caa4ee: Retrying in 19 seconds
    802254caa4ee: Retrying in 18 seconds
    802254caa4ee: Retrying in 17 seconds
    802254caa4ee: Retrying in 16 seconds
    802254caa4ee: Retrying in 15 seconds
    802254caa4ee: Retrying in [14](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:15) seconds
    802254caa4ee: Retrying in 13 seconds
    802254caa4ee: Retrying in 12 seconds
    802254caa4ee: Retrying in 11 seconds
    802254caa4ee: Retrying in 10 seconds
    802254caa4ee: Retrying in 9 seconds
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Downloading
    docker: unknown: <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJHMEUCIQCeIV4tfQcJ4orTMrZ5u4B8veMnlidVp99W1l6joTQRugIgDanE0IlIJAZ/aX61tAmHihmGywEMl7B[15](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:16)456kFL7MtIq7gQISxAAGgw3MTAwMTUwNDA4OTIiDGBDvMDxo9r+sdUzOCrLBIlSzKWaOEko/eQmED0w2OtilOeboB1PHAMnTlzh+Ru6CJ/sLjjwyQ5XsSNbDbWPezZZ/oKk2D8XEERqY+Xgma0H+PbLFkqROMKnNFtv7zvXwI7uYZdsov7YvSHnRZ+8KIztxvam+iBia3QAPgpqoP1Cv9mNw0kkB8aSMvjBNmqMyDkp7VJI+EJtFLnxTKQPDMeDR9eT4mFynBXuSMvoHNqoybqG7K2IRQA5wnhr/P6eZZ5xF3OySxo3P5Co9VeOkNOLpx2WaeZ+TqksZdFTjQML+j7MRi5NLDI7HUWP/P47APIbXrVaj4WFIs2BKvVXUeFwIJ+GxSsRu66YoXie53jnDWZiRAvfUOA2P3AKGYoVXlmwxN3UBDrbgFtUI6twAzH2UTBfviNoYYXHDa2EFjIQ8bLM1LdN554CCncJRzIheCBVDARw8+LIeT4zTJCRPJ4cdSL2QOFvYQiN8m3nEn09rMAUXOMRfGnvMXVkHC4EUtDtZUrZxKdHvA9MzqtVKMfYV3NPJtqxF9nxBqu4GxUIVhKTBozZtTMJtfDV2QsuwgvT3ZpY0Vr3qtBOnxGZ/cqwqGnCEjiYcIg43wSxmrUDR0/9StBCR4Fb1QTUqGcBVScT2R5oE5CKLoFxXkcCx8FNYz5B8Gqj0JCfFuKc/gUZQdCBMDPTFO8cp5brNfqimGtwfzaleQ65Co/4/HdlUGLWT4xdcEPPzxFSc/XCy0C7XefOhCPyc9Tu+FeoJZ6GTaI5YxcdtQHn+282ScMZA61LyiQ6NIMX2zR3MJyz8LsGOpoBWqE7+eLUXpfwp/Im0R2QzUeZdR85rxggo2vfOIG+jpJtye5KUxhxKvKN5pIiFOnnVADVhQNskCdF1SyntDNBK4IqE41480C/LdFyPuI362qp3EIVZ/aamajzroZUfw8EDyx7c+PEn6iz2CbJNKKpfFcvxMKM4yxVhFUWZdr/Xig10rH8E/KCLFsvAEO4jljVwNgxJZpLq51fqw==</Token-0><RequestId>XXTBWVERT9Z1XXV5</RequestId><HostId>asY4icJxCgIPWPkzZkKvjdKhjE0Ukw4Hmr1mihJ7J7ah+7z8a8mg3h0sabPaUrepodnjw471hbM=</HostId></Error>.
    See 'docker run --help'.
  
  Work dir:
    /home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c[17](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:18)38a3e7a1cd8a9ce8f488ab570/work/9c/0e0866d01d98ff46588f1ba7004635
  
  Container:
    wave.seqera.io/wt/5ac75ac6191e/rocker/tidyverse:4.4.1
  
  Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
  
   -- Check '/home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/meta/nextflow.log' file for details
  Execution cancelled -- Finishing pending tasks before exit
  Nextflow stderr:
  
  



FAILURE: Executed 1 tests in 390.406s (1 failed)

harmonbhasin · 2025-01-07T20:07:33Z

@harmonbhasin This is the error I get when nf-test is run on github. nf-test works locally. Can you let me know if you see what might be wrong here?

Otherwise I adopted your suggestions. Once the nf-test thing is resolved this is good for review by Will.

Run nf-test test --tag run --verbose

🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr


End-to-end test of MGS workflow

  Test [4ae1c173] 'Test run workflow' 
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/home/runner/work/mgs-workflow/mgs-workflow/main.nf` [admiring_stallman] DSL2 - revision: 73db19e0f9
    > Downloading plugin [email protected]
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
    > Downloading plugin [email protected]
    > [94/67ea50] Submitted process > RUN:RAW:QC:FASTQC_LABELED (1)
    > [d0/3647f3] Submitted process > RUN:CLEAN:FILTER_READS (1)
    > [9f/eb6025] Submitted process > RUN:RAW:QC:MULTIQC_LABELED (1)
    > [24/d22e6[8](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:9)] Submitted process > RUN:CLEAN:QC:FASTQC_LABELED (1)
    > [5d/1c48d5] Submitted process > RUN:PROFILE:SUBSET_READS_TARGET (1)
    > [54/1f6453] Submitted process > RUN:CLEAN:QC:MULTIQC_LABELED (1)
    > Staging foreign file: s3://nao-testing/index-test/output/results/ribo-ref-concat.fasta.gz
    > [87/1fc755] Submitted process > RUN:EXTRACT_VIRAL_READS:BBDUK_HITS (1)
    > [6e/d760af] Submitted process > RUN:PROFILE:BBDUK (1)
    > [64/fdca31] Submitted process > RUN:CLEAN:QC:SUMMARIZE_MULTIQC (1)
    > [cf/8dea48] Submitted process > RUN:RAW:QC:SUMMARIZE_MULTIQC (1)
    > [76/f[9](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:10)1a7d] Submitted process > RUN:EXTRACT_RAW_READS_FROM_PROCESSED (1)
    > [40/942af5] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:BBMERGE (1)
    > [19/fa812b] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:BBMERGE (1)
    > [6f/0589c1] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:SUMMARIZE_BBMERGE (1)
    > [5c/b45ec3] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:JOIN_FASTQ (1)
    > [ae/b4ea7a] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:SUMMARIZE_BBMERGE (1)
    > [ad/f49455] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:JOIN_FASTQ (1)
    > [61/38d663] Submitted process > RUN:EXTRACT_VIRAL_READS:CUTADAPT (1)
    > Staging foreign file: s3://nao-testing/index-test/output/results/kraken_db
    > [9c/0e0866] Submitted process > RUN:RAW:QC:MERGE_MULTIQC_BASIC
    > [b3/6a0269] Submitted process > RUN:CLEAN:QC:MERGE_MULTIQC_QBASE
    > ERROR ~ Error executing process > 'RUN:RAW:QC:MERGE_MULTIQC_BASIC'
    > 
    > Caused by:
    >   Process `RUN:RAW:QC:MERGE_MULTIQC_BASIC` terminated with an error exit status (125)
    > 
    > 
    > Command executed:
    > 
    >   #!/usr/bin/env Rscript
    >   library(tidyverse)
    >   in_paths <- str_split("raw_concat_gold_standard_qc_basic_stats.tsv.gz", " ")[[1]]
    >   print(in_paths)
    >   tabs <- lapply(in_paths, function(t) read_tsv(t, col_names = TRUE, cols(.default="c")))
    >   for (t in tabs) print(dim(t))
    >   tab_out <- bind_rows(tabs)
    >   print(dim(tab_out))
    >   sapply(tabs, nrow) %>% sum %>% print
    >   write_tsv(tab_out, "raw_concat_qc_basic_stats.tsv.gz")
    > 
    > Command exit status:
    >   125
    > 
    > Command output:
    >   (empty)
    > 
    > Command error:
    >   c0e4127c6ef4: Retrying in 1 second
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   c0e4127c6ef4: Downloading
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Downloading
    >   802254caa4ee: Retrying in 15 seconds
    >   802254caa4ee: Retrying in 14 seconds
    >   802254caa4ee: Retrying in 13 seconds
    >   802254caa4ee: Retrying in 12 seconds
    >   802254caa4ee: Retrying in 11 seconds
    >   802254caa4ee: Retrying in [10](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:11) seconds
    >   802254caa4ee: Retrying in 9 seconds
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Retrying in 20 seconds
    >   802254caa4ee: Retrying in 19 seconds
    >   802254caa4ee: Retrying in 18 seconds
    >   802254caa4ee: Retrying in 17 seconds
    >   802254caa4ee: Retrying in 16 seconds
    >   802254caa4ee: Retrying in 15 seconds
    >   802254caa4ee: Retrying in 14 seconds
    >   802254caa4ee: Retrying in 13 seconds
    >   802254caa4ee: Retrying in 12 seconds
    >   802254caa4ee: Retrying in [11](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:12) seconds
    >   802254caa4ee: Retrying in 10 seconds
    >   802254caa4ee: Retrying in 9 seconds
    >   802254caa4ee: Retrying in 8 seconds
    >   802254caa4ee: Retrying in 7 seconds
    >   802254caa4ee: Retrying in 6 seconds
    >   802254caa4ee: Retrying in 5 seconds
    >   802254caa4ee: Retrying in 4 seconds
    >   802254caa4ee: Retrying in 3 seconds
    >   802254caa4ee: Retrying in 2 seconds
    >   802254caa4ee: Retrying in 1 second
    >   802254caa4ee: Downloading
    >   docker: unknown: <?xml version="1.0" encoding="UTF-8"?>
    >   <Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJHMEUCIQCeIV4tfQcJ4orTMrZ5u4B8veMnlidVp99W1l6joTQRugIgDanE0IlIJAZ/aX61tAmHihmGywEMl7B15456kFL7MtIq7gQISxAAGgw3MTAwMTUwNDA4OTIiDGBDvMDxo9r+sdUzOCrLBIlSzKWaOEko/eQmED0w2OtilOeboB1PHAMnTlzh+Ru6CJ/sLjjwyQ5XsSNbDbWPezZZ/oKk2D8XEERqY+Xgma0H+PbLFkqROMKnNFtv7zvXwI7uYZdsov7YvSHnRZ+8KIztxvam+iBia3QAPgpqoP1Cv9mNw0kkB8aSMvjBNmqMyDkp7VJI+EJtFLnxTKQPDMeDR9eT4mFynBXuSMvoHNqoybqG7K2IRQA5wnhr/P6eZZ5xF3OySxo3P5Co9VeOkNOLpx2WaeZ+TqksZdFTjQML+j7MRi5NLDI7HUWP/P47APIbXrVaj4WFIs2BKvVXUeFwIJ+GxSsRu66YoXie53jnDWZiRAvfUOA2P3AKGYoVXlmwxN3UBDrbgFtUI6twAzH2UTBfviNoYYXHDa2EFjIQ8bLM1LdN554CCncJRzIheCBVDARw8+LIeT4zTJCRPJ4cdSL2QOFvYQiN8m3nEn09rMAUXOMRfGnvMXVkHC4EUtDtZUrZxKdHvA9MzqtVKMfYV3NPJtqxF9nxBqu4GxUIVhKTBozZtTMJtfDV2QsuwgvT3ZpY0Vr3qtBOnxGZ/cqwqGnCEjiYcIg43wSxmrUDR0/9StBCR4Fb1QTUqGcBVScT2R5oE5CKLoFxXkcCx8FNYz5B8Gqj0JCfFuKc/gUZQdCBMDPTFO8cp5brNfqimGtwfzaleQ65Co/4/HdlUGLWT4xdcEPPzxFSc/XCy0C7XefOhCPyc9Tu+FeoJZ6GTaI5YxcdtQHn+282ScMZA61LyiQ6NIMX2zR3MJyz8LsGOpoBWqE7+eLUXpfwp/Im0R2QzUeZdR85rxggo2vfOIG+jpJtye5KUxhxKvKN5pIiFOnnVADVhQNskCdF1SyntDNBK4IqE41480C/LdFyPuI362qp3EIVZ/aamajzroZUfw8EDyx7c+PEn6iz2CbJNKKpfFcvxMKM4yxVhFUWZdr/Xig10rH8E/KCLFsvAEO4jljVwNgxJZpLq51fqw==</Token-0><RequestId>XXTBWVERT9Z1XXV5</RequestId><HostId>asY4icJxCgIPWPkzZkKvjdKhjE0Ukw4Hmr1mihJ7J7ah+7z8a8mg3h0sabPaUrepodnjw471hbM=</HostId></Error>.
    >   See 'docker run --help'.
    > 
    > Work dir:
    >   /home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/work/9c/0e0866d01d98ff46588f1ba7004635
    > 
    > Container:
    >   wave.seqera.io/wt/5ac75ac6191e/rocker/tidyverse:4.4.1
    > 
    > Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
    > 
    >  -- Check '/home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/meta/nextflow.log' file for details
    > Execution cancelled -- Finishing pending tasks before exit
    FAILED (390.383s)

  Assertion failed: 
  
  assert workflow.success
         |        |
         workflow false
  
  Nextflow stdout:
  
  N E X T F L O W  ~  version 24.10.3
  Launching `/home/runner/work/mgs-workflow/mgs-workflow/main.nf` [admiring_stallman] DSL2 - revision: 73db19e0f9
  Downloading plugin [email protected]
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  WARN: WORKFLOW OUTPUT DSL IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
  Downloading plugin [email protected]
  [94/67ea50] Submitted process > RUN:RAW:QC:FASTQC_LABELED (1)
  [d0/3647f3] Submitted process > RUN:CLEAN:FILTER_READS (1)
  [9f/eb6025] Submitted process > RUN:RAW:QC:MULTIQC_LABELED (1)
  [24/d22e68] Submitted process > RUN:CLEAN:QC:FASTQC_LABELED (1)
  [5d/1c48d5] Submitted process > RUN:PROFILE:SUBSET_READS_TARGET (1)
  [54/1f6453] Submitted process > RUN:CLEAN:QC:MULTIQC_LABELED (1)
  Staging foreign file: s3://nao-testing/index-test/output/results/ribo-ref-concat.fasta.gz
  [87/1fc755] Submitted process > RUN:EXTRACT_VIRAL_READS:BBDUK_HITS (1)
  [6e/d760af] Submitted process > RUN:PROFILE:BBDUK (1)
  [64/fdca31] Submitted process > RUN:CLEAN:QC:SUMMARIZE_MULTIQC (1)
  [cf/8dea48] Submitted process > RUN:RAW:QC:SUMMARIZE_MULTIQC (1)
  [76/f91a7d] Submitted process > RUN:EXTRACT_RAW_READS_FROM_PROCESSED (1)
  [40/942af5] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:BBMERGE (1)
  [19/fa8[12](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:13)b] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:BBMERGE (1)
  [6f/0589c1] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:SUMMARIZE_BBMERGE (1)
  [5c/b45ec3] Submitted process > RUN:PROFILE:TAXONOMY_NORIBO:JOIN_FASTQ (1)
  [ae/b4ea7a] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:SUMMARIZE_BBMERGE (1)
  [ad/f49455] Submitted process > RUN:PROFILE:TAXONOMY_RIBO:JOIN_FASTQ (1)
  [61/38d663] Submitted process > RUN:EXTRACT_VIRAL_READS:CUTADAPT (1)
  Staging foreign file: s3://nao-testing/index-test/output/results/kraken_db
  [9c/0e0866] Submitted process > RUN:RAW:QC:MERGE_MULTIQC_BASIC
  [b3/6a0269] Submitted process > RUN:CLEAN:QC:MERGE_MULTIQC_QBASE
  ERROR ~ Error executing process > 'RUN:RAW:QC:MERGE_MULTIQC_BASIC'
  
  Caused by:
    Process `RUN:RAW:QC:MERGE_MULTIQC_BASIC` terminated with an error exit status (125)
  
  
  Command executed:
  
    #!/usr/bin/env Rscript
    library(tidyverse)
    in_paths <- str_split("raw_concat_gold_standard_qc_basic_stats.tsv.gz", " ")[[1]]
    print(in_paths)
    tabs <- lapply(in_paths, function(t) read_tsv(t, col_names = TRUE, cols(.default="c")))
    for (t in tabs) print(dim(t))
    tab_out <- bind_rows(tabs)
    print(dim(tab_out))
    sapply(tabs, nrow) %>% sum %>% print
    write_tsv(tab_out, "raw_concat_qc_basic_stats.tsv.gz")
  
  Command exit status:
    125
  
  Command output:
    (empty)
  
  Command error:
    c0e4127c6ef4: Retrying in 1 second
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    c0e4127c6ef4: Downloading
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Downloading
    802254caa4ee: Retrying in 15 seconds
    802254caa4ee: Retrying in 14 seconds
    802254caa4ee: Retrying in [13](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:14) seconds
    802254caa4ee: Retrying in 12 seconds
    802254caa4ee: Retrying in 11 seconds
    802254caa4ee: Retrying in 10 seconds
    802254caa4ee: Retrying in 9 seconds
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Retrying in 20 seconds
    802254caa4ee: Retrying in 19 seconds
    802254caa4ee: Retrying in 18 seconds
    802254caa4ee: Retrying in 17 seconds
    802254caa4ee: Retrying in 16 seconds
    802254caa4ee: Retrying in 15 seconds
    802254caa4ee: Retrying in [14](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:15) seconds
    802254caa4ee: Retrying in 13 seconds
    802254caa4ee: Retrying in 12 seconds
    802254caa4ee: Retrying in 11 seconds
    802254caa4ee: Retrying in 10 seconds
    802254caa4ee: Retrying in 9 seconds
    802254caa4ee: Retrying in 8 seconds
    802254caa4ee: Retrying in 7 seconds
    802254caa4ee: Retrying in 6 seconds
    802254caa4ee: Retrying in 5 seconds
    802254caa4ee: Retrying in 4 seconds
    802254caa4ee: Retrying in 3 seconds
    802254caa4ee: Retrying in 2 seconds
    802254caa4ee: Retrying in 1 second
    802254caa4ee: Downloading
    docker: unknown: <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJHMEUCIQCeIV4tfQcJ4orTMrZ5u4B8veMnlidVp99W1l6joTQRugIgDanE0IlIJAZ/aX61tAmHihmGywEMl7B[15](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:16)456kFL7MtIq7gQISxAAGgw3MTAwMTUwNDA4OTIiDGBDvMDxo9r+sdUzOCrLBIlSzKWaOEko/eQmED0w2OtilOeboB1PHAMnTlzh+Ru6CJ/sLjjwyQ5XsSNbDbWPezZZ/oKk2D8XEERqY+Xgma0H+PbLFkqROMKnNFtv7zvXwI7uYZdsov7YvSHnRZ+8KIztxvam+iBia3QAPgpqoP1Cv9mNw0kkB8aSMvjBNmqMyDkp7VJI+EJtFLnxTKQPDMeDR9eT4mFynBXuSMvoHNqoybqG7K2IRQA5wnhr/P6eZZ5xF3OySxo3P5Co9VeOkNOLpx2WaeZ+TqksZdFTjQML+j7MRi5NLDI7HUWP/P47APIbXrVaj4WFIs2BKvVXUeFwIJ+GxSsRu66YoXie53jnDWZiRAvfUOA2P3AKGYoVXlmwxN3UBDrbgFtUI6twAzH2UTBfviNoYYXHDa2EFjIQ8bLM1LdN554CCncJRzIheCBVDARw8+LIeT4zTJCRPJ4cdSL2QOFvYQiN8m3nEn09rMAUXOMRfGnvMXVkHC4EUtDtZUrZxKdHvA9MzqtVKMfYV3NPJtqxF9nxBqu4GxUIVhKTBozZtTMJtfDV2QsuwgvT3ZpY0Vr3qtBOnxGZ/cqwqGnCEjiYcIg43wSxmrUDR0/9StBCR4Fb1QTUqGcBVScT2R5oE5CKLoFxXkcCx8FNYz5B8Gqj0JCfFuKc/gUZQdCBMDPTFO8cp5brNfqimGtwfzaleQ65Co/4/HdlUGLWT4xdcEPPzxFSc/XCy0C7XefOhCPyc9Tu+FeoJZ6GTaI5YxcdtQHn+282ScMZA61LyiQ6NIMX2zR3MJyz8LsGOpoBWqE7+eLUXpfwp/Im0R2QzUeZdR85rxggo2vfOIG+jpJtye5KUxhxKvKN5pIiFOnnVADVhQNskCdF1SyntDNBK4IqE41480C/LdFyPuI362qp3EIVZ/aamajzroZUfw8EDyx7c+PEn6iz2CbJNKKpfFcvxMKM4yxVhFUWZdr/Xig10rH8E/KCLFsvAEO4jljVwNgxJZpLq51fqw==</Token-0><RequestId>XXTBWVERT9Z1XXV5</RequestId><HostId>asY4icJxCgIPWPkzZkKvjdKhjE0Ukw4Hmr1mihJ7J7ah+7z8a8mg3h0sabPaUrepodnjw471hbM=</HostId></Error>.
    See 'docker run --help'.
  
  Work dir:
    /home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c[17](https://github.com/naobservatory/mgs-workflow/actions/runs/12638560900/job/35215087099?pr=133#step:6:18)38a3e7a1cd8a9ce8f488ab570/work/9c/0e0866d01d98ff46588f1ba7004635
  
  Container:
    wave.seqera.io/wt/5ac75ac6191e/rocker/tidyverse:4.4.1
  
  Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
  
   -- Check '/home/runner/work/mgs-workflow/mgs-workflow/.nf-test/tests/4ae1c1738a3e7a1cd8a9ce8f488ab570/meta/nextflow.log' file for details
  Execution cancelled -- Finishing pending tasks before exit
  Nextflow stderr:
  
  



FAILURE: Executed 1 tests in 390.406s (1 failed)

@simonleandergrimm specific docker images from sequera (and even dockerhub for that matter) have given me trouble on Github Actions in the past, and my guess is that you're having a similar issue, although I have not run into this specific issue before. As of right now, the best way I know of fixing issues with docker images issues related to Github Actions is trial-and-error. The first thing I would do is try a different distribution of tidyverse, this might be a good place to start, https://hub.docker.com/r/rocker/tidyverse.

harmonbhasin

This all looks good to me! Once the docker image issue is fixed, it'll be ready for Will.

harmonbhasin

Thanks for fixing the issue, this is good for you to look at @willbradshaw.

willbradshaw · 2025-01-14T17:31:54Z

@simonleandergrimm based on our discussion yesterday, are you planning on adding any tests before we merge this?

simonleandergrimm · 2025-01-15T14:05:17Z

@willbradshaw I likely do not have time for this in the next few days (maybe even not until I leave for the holiday, due to Zephyr and P2RA manuscript demands).

willbradshaw · 2025-01-15T23:39:35Z

OK, let's drop that here then. I do want my comment on configs/read_type.config to be addressed before I merge this, though.

…arameter for data being ONT. Added adapters to the ONT file so it doesn't break.

simonleandergrimm · 2025-01-16T00:09:26Z

OK, let's drop that here then. I do want my comment on configs/read_type.config to be addressed before I merge this, though.

I dropped the ont param from read_type.config, and added the following bit to the run config files. Let me know if the wording looks good.

 // Sequencing platform
    ont = false // Whether the sequencing is ONT (true) or Illumina (false)

Still waiting for checks to run to completion.

simonleandergrimm · 2025-01-16T00:25:49Z

Ah looks like the ONT test files can't get fetched due to being on restricted. Will fix shortly.

simonleandergrimm · 2025-01-21T01:14:41Z

@willbradshaw This now passes tests. I also adapted the code to fit the newest structure of dev. Can you review it and let me know if I can merge it?

configs/read_type.config

willbradshaw · 2025-01-21T18:50:02Z

@simonleandergrimm code looks fine, but there are conflicts with dev that need to be resolved before I can merge this. You need to merge dev into this branch and fix any conflicts.

simonleandergrimm · 2025-01-21T22:00:04Z

@simonleandergrimm code looks fine, but there are conflicts with dev that need to be resolved before I can merge this. You need to merge dev into this branch and fix any conflicts.

Done!

simonleandergrimm and others added 6 commits December 9, 2024 16:17

Added variable that checks if data is ONT, based on fastq file name.

4fbed83

Adding the filtlong process and docker image,

71757bc

Added code to pull out legnth data from multiqc.

66569ad

Adding filtlong to the Cleaning step.

09eaf8f

Merge branch 'single-read-raw-clean' into ont-raw-clean

d925140

Update main.nf

2e039df

simonleandergrimm changed the base branch from master to single-read-raw-clean December 18, 2024 20:42

simonleandergrimm added 7 commits December 18, 2024 20:45

Merge branch 'simon-ont-raw-clean' of https://github.com/naobservator…

bade5a6

…y/mgs-workflow into simon-ont-raw-clean

Merge branch 'single-read-raw-clean' into simon-ont-raw-clean

2f56645

edited FASTP logic to take into account fastp_single and fastp_paired

e145901

Renamed summarize_multiqc processes/folders/scripts to not have pair …

34793cb

…in the name.

Adopting testing set up from profile ont branch

22073c1

Fixed summarize multiqc typo and uneeded single-end variable for fastp

3ce9cc6

dropped unneeded brackets.

cfd0ddb

simonleandergrimm requested a review from harmonbhasin December 18, 2024 21:34

simonleandergrimm marked this pull request as ready for review December 18, 2024 21:35

simonleandergrimm changed the title ~~Simon ont raw clean~~ Adding ONT functionality to RAW and CLEAN Dec 18, 2024

harmonbhasin approved these changes Dec 19, 2024

View reviewed changes

modules/local/summarizeMultiqc/main.nf Show resolved Hide resolved

harmonbhasin requested changes Dec 19, 2024

View reviewed changes

Base automatically changed from single-read-raw-clean to dev December 20, 2024 13:55

simonleandergrimm added 2 commits December 20, 2024 15:32

Added test config for ONT

bbd108b

Added ONT run to dev

c9b72c4

Merge branch 'dev' into simon-ont-raw-clean

c676b5d

simonleandergrimm assigned harmonbhasin Jan 6, 2025

simonleandergrimm requested a review from harmonbhasin January 6, 2025 19:47

harmonbhasin requested changes Jan 7, 2025

View reviewed changes

harmonbhasin mentioned this pull request Jan 10, 2025

Update run_validation.config (dropping read_type.config) #140

Merged

harmonbhasin previously approved these changes Jan 10, 2025

View reviewed changes

willbradshaw assigned simonleandergrimm and unassigned willbradshaw Jan 14, 2025

simonleandergrimm and others added 3 commits January 15, 2025 23:50

Dropping ont from read_type.config

7a41916

Update run_validation.config

1df4699

Fixed tag in end-to-end-se.yml for the ont test run. Added a manual p…

0f5f68d

…arameter for data being ONT. Added adapters to the ONT file so it doesn't break.

simonleandergrimm dismissed harmonbhasin’s stale review via 0f5f68d January 16, 2025 00:07

Merge branch 'dev' into simon-ont-raw-clean

93c50ed

simonleandergrimm added 5 commits January 17, 2025 20:26

moved location of ONT WW test data

919b3c4

Merge branch 'dev' into simon-ont-raw-clean

37fd2f9

Adding qc lengths to runQc. Adding filtlong to subsetTrim.

58416f0

fixed multiqc output styling.

2224711

fixed address of ont ww test data.

b6868c1

Increased fitlong mean quality score to 99%

0b9672d

willbradshaw previously approved these changes Jan 21, 2025

View reviewed changes

configs/read_type.config Outdated Show resolved Hide resolved

missed adding ont param to tests/run.config. now fixed.

7fd1ee5

simonleandergrimm dismissed willbradshaw’s stale review via 7fd1ee5 January 21, 2025 19:12

Merge branch 'dev' into simon-ont-raw-clean

0eeabe3

willbradshaw approved these changes Jan 22, 2025

View reviewed changes

willbradshaw merged commit 3c07823 into dev Jan 22, 2025
5 checks passed

willbradshaw deleted the simon-ont-raw-clean branch January 22, 2025 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ONT functionality to RAW and CLEAN #133

Adding ONT functionality to RAW and CLEAN #133

simonleandergrimm commented Dec 18, 2024 •

edited

Loading

simonleandergrimm commented Dec 18, 2024

harmonbhasin left a comment

harmonbhasin left a comment •

edited

Loading

simonleandergrimm commented Jan 6, 2025

simonleandergrimm commented Jan 6, 2025 •

edited

Loading

harmonbhasin commented Jan 7, 2025

harmonbhasin left a comment

harmonbhasin left a comment

willbradshaw commented Jan 14, 2025

simonleandergrimm commented Jan 15, 2025

willbradshaw commented Jan 15, 2025

simonleandergrimm commented Jan 16, 2025 •

edited

Loading

simonleandergrimm commented Jan 16, 2025

simonleandergrimm commented Jan 21, 2025

willbradshaw commented Jan 21, 2025

simonleandergrimm commented Jan 21, 2025

Adding ONT functionality to RAW and CLEAN #133

Adding ONT functionality to RAW and CLEAN #133

Conversation

simonleandergrimm commented Dec 18, 2024 • edited Loading

simonleandergrimm commented Dec 18, 2024

harmonbhasin left a comment

Choose a reason for hiding this comment

harmonbhasin left a comment • edited Loading

Choose a reason for hiding this comment

simonleandergrimm commented Jan 6, 2025

simonleandergrimm commented Jan 6, 2025 • edited Loading

harmonbhasin commented Jan 7, 2025

harmonbhasin left a comment

Choose a reason for hiding this comment

harmonbhasin left a comment

Choose a reason for hiding this comment

willbradshaw commented Jan 14, 2025

simonleandergrimm commented Jan 15, 2025

willbradshaw commented Jan 15, 2025

simonleandergrimm commented Jan 16, 2025 • edited Loading

simonleandergrimm commented Jan 16, 2025

simonleandergrimm commented Jan 21, 2025

willbradshaw commented Jan 21, 2025

simonleandergrimm commented Jan 21, 2025

simonleandergrimm commented Dec 18, 2024 •

edited

Loading

harmonbhasin left a comment •

edited

Loading

simonleandergrimm commented Jan 6, 2025 •

edited

Loading

simonleandergrimm commented Jan 16, 2025 •

edited

Loading