feat: set intermediate and final output files

README.md

@@ -12,6 +12,7 @@ _MIRFLOWZ_ is a [Snakemake][snakemake] workflow for mapping miRNAs and isomiRs.
 2. [Usage](#usage)
     - [Preparing inputs](#preparing-inputs)
     - [Running the workflow](#running-the-workflow)
+    - [Expected output files](#expected-output-files)
     - [Creating a Snakemake report](#creating-a-snakemake-report)
 3. [Workflow description](#workflow-description)
 4. [Contributing](#contributing)
@@ -251,6 +252,50 @@ snakemake \
 After successful execution of the workflow, results and logs will be found in
 the `results/` and `logs/` directories, respectively.
 
+### Expected output files
+
+Upon successful execution of _MIRFLOWZ_, the tool automatically removes all
+intermediate files generated during the process. The final output comprises:
+
+1. A SAM file containing alignments intersecting a pri-miR locus. These
+alignments intersect with extended start and/or end positions specified in the
+provided pri-miR annotations. Please note that they may not contribute to the
+final counting and will not appear in the final table.
+
+2. A SAM file containing alignments intersecting a miRNA locus. Similar to the
+previous file, these alignments intersect with extended start and/or end
+positions specified in the provided miRNA annotations. They may not contribute
+to the final counting and might be absent from the final table.
+
+3. A SAM file containing the uncollapsed set of alignments that contribute to
+the final counting.
+
+4. A BAM file containing the uncollapsed set of alignments contributing to the
+final counting and its corresponding index file (`bam..bai`).
+
+5. Table(s) containing the counting data from all libraries for (iso)miRs
+and/or pri-miRs. Each row corresponds to a miRNA species, and each column
+represents a sample library. Counting involves aggregating contributions from
+all alignments, calculated as the ratio of collapsed reads in th alignment to
+the number of hits (NH value).
+
+To retain all intermediate files, include --no-hooks in the workflow call.
+
+```bash
+snakemake \
+    --snakefile="path/to/Snakefile" \
+    --cores 4  \
+    --configfile="path/to/config.yaml" \
+    --use-conda \
+    --printshellcmds \
+    --rerun-incomplete \
+    --no-hooks \
+    --verbose
+```
+
+After successful execution of the workflow, the intermediate files will be
+found in the `results/inter_files` directory.
+
 ### Creating a Snakemake report
 
 Snakemake provides the option to generate a detailed HTML report on runtime

config/config_schema.json

@@ -30,6 +30,11 @@
             "default": "results/",
             "description": "Path to the output directory."
         },
+        "tmp_dir":{
+            "type": "string",
+            "default": "results/inter_files",
+            "description": "Path to the temporary directory storing the intermediate files."
+        },
         "local_log":{
             "type": "string",
             "default": "logs/local/",

config/config_template.yaml

@@ -32,6 +32,7 @@ map_chr_file: path/to/ucsc_ensembl_mappings.tsv
 #### DIRECTORIES ####
 
 output_dir: results/
+tmp_dir: results/inter_files
 local_log: logs/local/
 cluster_log: logs/cluster/
 scripts_dir: ../scripts/

test/expected_output.md5

@@ -1,58 +1,58 @@
 68f943f89b52d628851dd97fb1399d68  results/TABLES/all_mirna_counts.tab
-eec9be6cda61d2728290c92c1209f455  results/TABLES/mirna_counts_test_lib
 363ecee318c57ee7e2e45ca468007baa  results/TABLES/all_pri-mir_counts.tab
-a844e3a29159e36e2f17a0646d1e8c5f  results/TABLES/pri-mir_counts_test_lib
 0d76977b2e36046cc176112776c5fa4e  results/test_lib/alignments_intersecting_mirna_uncollapsed_sorted.bam.bai
-36f7d024fe6ddfd3e788aebf61c61061  results/test_lib/oligomap_genome_sorted.fasta
-48e605df55bf2dd37ea5a5a74eb5872a  results/test_lib/mappings_all.sam
-d41d8cd98f00b204e9800998ecf8427e  results/test_lib/oligomap_transcriptome_mappings.fasta
-eea903fc0ab81054cf8e34193f80f4a7  results/test_lib/mappings_all_removed_inferiors.sam
-98498ac521f451426a9dbabcbecb5f25  results/test_lib/alignments_intersecting_primir.bam
-defdc8c46e1d73692edde0e0278f2d5e  results/test_lib/oligomap_genome_mappings.fasta
-1649738f226e8979d4d88a3ae47fa423  results/test_lib/segemehl_transcriptome_mappings.sam
-9ecee9ab80daba0a53076b05c9f6ff53  results/test_lib/alignments_intersecting_mirna_uncollapsed_sorted.bam
-1649738f226e8979d4d88a3ae47fa423  results/test_lib/transcriptome_mappings_filtered_nh.sam
-8e22ddfa7c39ce7e4ec5945dff1576ef  results/test_lib/alignments_all.bam
-a124a5afdb5f7bfbcc5683260556c9c4  results/test_lib/mappings_all_no_header.sam
-dd00dea3549dc1ad14f9e1505d397de5  results/test_lib/alignments_all.sam
-8c24d619073f4c5ca1f439fe429d0ef4  results/test_lib/alignments_intersecting_mirna_tag.sam
-d41d8cd98f00b204e9800998ecf8427e  results/test_lib/oligomap_transcriptome_sorted.fasta
-c218718d93f48e5987fc18b33dc488f0  results/test_lib/segemehl_genome_mappings.sam
-d41d8cd98f00b204e9800998ecf8427e  results/test_lib/transcriptome_mappings_to_genome.sam
-63a32839360a985b68e0685aafad5c54  results/test_lib/fa/reads.fa
-5cc557ec2073144f47fe28ac145f4869  results/test_lib/alignments_intersecting_mirna_uncollapsed.sam
-edcb854702519c0002d8ce89a21e54ef  results/test_lib/reads_formatted.fasta
-1a547487b8e92ad85bb26ff9b1db1f93  results/test_lib/intersected_extended_mirna.bed
-721071f3ead528aa71978508db8d73f9  results/test_lib/alignments_all_sorted_test_lib.bam
-ec0e9bcc8ea857da897035c8fca4078f  results/test_lib/reads_trimmed_adapters.fasta
-bbfc27c84b66ff41bfeee73f701b4b29  results/test_lib/alignments_intersecting_mirna_uncollapsed.bam
-81bed7fc879f7a16c12d2ba912263c46  results/test_lib/alignments_intersecting_mirna.sam
-dd560414078330bf3138f039da109093  results/test_lib/genome_mappings.sam
-f5cb65466d328036a15b66cfbd4d8419  results/test_lib/oligomap_genome_report.txt
-6cbdb9299e09b3e39b79a50db69226b5  results/test_lib/transcriptome_mappings_no_header.sam
-1649738f226e8979d4d88a3ae47fa423  results/test_lib/transcriptome_mappings.sam
-947607be69c16246f8dc9adbd9b971c8  results/test_lib/oligomap_genome_mappings.sam
-9833208a79143eaf3f2a5fdeca0b2d94  results/test_lib/alignments_intersecting_mirna_sorted_tag.sam
-02096523b293082629d5b895085468a3  results/test_lib/alignments_intersecting_primir_sorted.bam
-d41d8cd98f00b204e9800998ecf8427e  results/test_lib/oligomap_transcriptome_mappings.sam
-a124a5afdb5f7bfbcc5683260556c9c4  results/test_lib/genome_mappings_no_header.sam
-dd560414078330bf3138f039da109093  results/test_lib/genome_mappings_filtered_nh.sam
-ae4c4963ca2cd206952b2ea2c58301dd  results/test_lib/mappings_all_sorted_by_id.sam
-2c77ffa021dda190d82f3f54a3312393  results/test_lib/reads_collapsed.fasta
-f68693cfaa1e6ea78e1a5562ade6d9ed  results/test_lib/intersected_extended_primir.bed
-61f12595db9421926073d6675f7c3c42  results/test_lib/alignments_intersecting_primir.sam
-c2a5770a755ada66ef63d96eec4afb00  results/test_lib/reads_filtered_for_oligomap.fasta
-fe5388094985e9604a302d39d2abc82c  results/test_lib/oligomap_transcriptome_report.txt
-be7a0d92e57480190de57eb30baffa36  results/extended_mirna_annotation_6_nt.gff3
-8148cd880602255be166beb59bbed95a  results/genome_header.sam
-09e24a504bfec37fee3d5ff1b5c7738e  results/exons.bed
-4fb453846e88593d0cac13220ec2d685  results/segemehl_genome_index.idx
-d34fc868b861b1bc46db07a397dc0f10  results/genome_processed.fa.fai
-21e102e4ebd3508bb06f46366a3d578d  results/exons.gtf
-003b92b245ac336e3d70a513033e1cee  results/transcriptome_trimmed_id.fa
-44dbf7c3eae00d0bc8d5e1319123746c  results/chr_size.txt
-cc5c3512dab0e269d82bd625de74198e  results/extended_primir_annotation_6_nt.gff3
-f28cc0143ab6659bef3de3a7afa1dccc  results/mirna_annotations.gff3
-2d437f8681f4248d4f2075f86debb920  results/transcriptome.fa
-7eb64c112830266bcf416ded60b4cf77  results/segemehl_transcriptome_index.idx
-4fba145540a2c61f29bfddfd0f5a4d4e  results/genome_processed.fa
+f91c144e491e447a50369a67220a832f  results/test_lib/alignments_intersecting_mirna_uncollapsed_sorted.bam
+a8b1a66aecf4d7b583362ea8619228ed  results/test_lib/alignments_intersecting_mirna_uncollapsed.sam
+9f0bad0ed3c62d0410060d8b332315e8  results/test_lib/alignments_intersecting_mirna.sam
+4ae56cdb8de0fbaac24b4a49d356f7f8  results/test_lib/alignments_intersecting_primir.sam
+eec9be6cda61d2728290c92c1209f455  results/inter_files/TABLES/mirna_counts_test_lib
+a844e3a29159e36e2f17a0646d1e8c5f  results/inter_files/TABLES/pri-mir_counts_test_lib
+36f7d024fe6ddfd3e788aebf61c61061  results/inter_files/test_lib/oligomap_genome_sorted.fasta
+48e605df55bf2dd37ea5a5a74eb5872a  results/inter_files/test_lib/mappings_all.sam
+d41d8cd98f00b204e9800998ecf8427e  results/inter_files/test_lib/oligomap_transcriptome_mappings.fasta
+f54bacf9bf4188541a0c0fedc203e3ed  results/inter_files/test_lib/mappings_all_removed_inferiors.sam
+4b86be9b7ed15ddc0067b8de4aad431c  results/inter_files/test_lib/alignments_intersecting_primir.bam
+defdc8c46e1d73692edde0e0278f2d5e  results/inter_files/test_lib/oligomap_genome_mappings.fasta
+3aca095999e737c5d9cdb66540e8b195  results/inter_files/test_lib/segemehl_transcriptome_mappings.sam
+3aca095999e737c5d9cdb66540e8b195  results/inter_files/test_lib/transcriptome_mappings_filtered_nh.sam
+698711937e6d98dd65b70b3a738388b4  results/inter_files/test_lib/alignments_all.bam
+a124a5afdb5f7bfbcc5683260556c9c4  results/inter_files/test_lib/mappings_all_no_header.sam
+cb542d2dd6b4405d690086de0bb5ec70  results/inter_files/test_lib/alignments_all.sam
+d8ab74abfa3ed2b2a92c83142af1c638  results/inter_files/test_lib/alignments_intersecting_mirna_tag.sam
+d41d8cd98f00b204e9800998ecf8427e  results/inter_files/test_lib/oligomap_transcriptome_sorted.fasta
+f34a0091f633db03a940d0c790ad265a  results/inter_files/test_lib/segemehl_genome_mappings.sam
+d41d8cd98f00b204e9800998ecf8427e  results/inter_files/test_lib/transcriptome_mappings_to_genome.sam
+63a32839360a985b68e0685aafad5c54  results/inter_files/test_lib/fa/reads.fa
+edcb854702519c0002d8ce89a21e54ef  results/inter_files/test_lib/reads_formatted.fasta
+1a547487b8e92ad85bb26ff9b1db1f93  results/inter_files/test_lib/intersected_extended_mirna.bed
+a71a2dd39c82baee52d5dbe2e3a39457  results/inter_files/test_lib/alignments_all_sorted_test_lib.bam
+ec0e9bcc8ea857da897035c8fca4078f  results/inter_files/test_lib/reads_trimmed_adapters.fasta
+acf1608593f39294e0137069f6351058  results/inter_files/test_lib/alignments_intersecting_mirna_uncollapsed.bam
+0454bc9f3edd9348a7b3e08d9c3007d8  results/inter_files/test_lib/genome_mappings.sam
+f5cb65466d328036a15b66cfbd4d8419  results/inter_files/test_lib/oligomap_genome_report.txt
+6cbdb9299e09b3e39b79a50db69226b5  results/inter_files/test_lib/transcriptome_mappings_no_header.sam
+3aca095999e737c5d9cdb66540e8b195  results/inter_files/test_lib/transcriptome_mappings.sam
+947607be69c16246f8dc9adbd9b971c8  results/inter_files/test_lib/oligomap_genome_mappings.sam
+fa14b33623fd12b068a6d4ae301e7f49  results/inter_files/test_lib/alignments_intersecting_mirna_sorted_tag.sam
+b6de7f5615b4b05834f4af11df993345  results/inter_files/test_lib/alignments_intersecting_primir_sorted.bam
+d41d8cd98f00b204e9800998ecf8427e  results/inter_files/test_lib/oligomap_transcriptome_mappings.sam
+a124a5afdb5f7bfbcc5683260556c9c4  results/inter_files/test_lib/genome_mappings_no_header.sam
+0454bc9f3edd9348a7b3e08d9c3007d8  results/inter_files/test_lib/genome_mappings_filtered_nh.sam
+09c89a2769c919e58c3a3d3cbe2ceaf6  results/inter_files/test_lib/mappings_all_sorted_by_id.sam
+2c77ffa021dda190d82f3f54a3312393  results/inter_files/test_lib/reads_collapsed.fasta
+f68693cfaa1e6ea78e1a5562ade6d9ed  results/inter_files/test_lib/intersected_extended_primir.bed
+c2a5770a755ada66ef63d96eec4afb00  results/inter_files/test_lib/reads_filtered_for_oligomap.fasta
+fe5388094985e9604a302d39d2abc82c  results/inter_files/test_lib/oligomap_transcriptome_report.txt
+be7a0d92e57480190de57eb30baffa36  results/inter_files/extended_mirna_annotation_6_nt.gff3
+8148cd880602255be166beb59bbed95a  results/inter_files/genome_header.sam
+09e24a504bfec37fee3d5ff1b5c7738e  results/inter_files/exons.bed
+4fb453846e88593d0cac13220ec2d685  results/inter_files/segemehl_genome_index.idx
+d34fc868b861b1bc46db07a397dc0f10  results/inter_files/genome_processed.fa.fai
+21e102e4ebd3508bb06f46366a3d578d  results/inter_files/exons.gtf
+003b92b245ac336e3d70a513033e1cee  results/inter_files/transcriptome_trimmed_id.fa
+44dbf7c3eae00d0bc8d5e1319123746c  results/inter_files/chr_size.txt
+cc5c3512dab0e269d82bd625de74198e  results/inter_files/extended_primir_annotation_6_nt.gff3
+f28cc0143ab6659bef3de3a7afa1dccc  results/inter_files/mirna_annotations.gff3
+2d437f8681f4248d4f2075f86debb920  results/inter_files/transcriptome.fa
+7eb64c112830266bcf416ded60b4cf77  results/inter_files/segemehl_transcriptome_index.idx
+4fba145540a2c61f29bfddfd0f5a4d4e  results/inter_files/genome_processed.fa

test/test_workflow_local_with_conda.sh

@@ -25,9 +25,9 @@ snakemake \
     --use-conda \
     --printshellcmds \
     --rerun-incomplete \
+    --no-hooks \
     --verbose
 
-
 # Snakemake report
 snakemake \
     --snakefile="../workflow/Snakefile" \

test/test_workflow_local_with_singularity.sh

@@ -26,9 +26,9 @@ snakemake \
     --singularity-args "--bind ${PWD}/../" \
     --printshellcmds \
     --rerun-incomplete \
+    --no-hooks \
     --verbose
 
-
 # Snakemake report
 snakemake \
     --snakefile="../workflow/Snakefile" \

test/test_workflow_slurm_with_conda.sh

@@ -41,6 +41,7 @@ snakemake \
     --use-conda \
     --printshellcmds \
     --rerun-incomplete \
+    --no-hooks \
     --verbose
 
 # Snakemake report

test/test_workflow_slurm_with_singularity.sh

@@ -42,6 +42,7 @@ snakemake \
     --singularity-args="--bind ${PWD}/../" \
     --printshellcmds \
     --rerun-incomplete \
+    --no-hooks \
     --verbose
 
 # Snakemake report

workflow/Snakefile

@@ -31,6 +31,23 @@ validate(config, Path("../config/config_schema.json"))
 
 
 OUT_DIR = Path(config["output_dir"])
+TMP_DIR = Path(config["tmp_dir"])
+LOG_DIR = Path(f"{config['local_log']}/../")
+
+
+###############################################################################
+### onSuccess/onError handlers configuration
+###############################################################################
+
+
+onsuccess:
+    print("\nWORKFLOW SUCCEED. Removing intermediate files.\n")
+    shell("rm -rf {TMP_DIR}")
+
+
+onerror:
+    print("\nWORKFLOW FAILED. Check the log file in the LOGS/ directory.\n")
+    shell("cat {log} > {LOG_DIR}/failed_workflow.log")
 
 
 ###############################################################################
@@ -67,14 +84,14 @@ rule finish:
             OUT_DIR / "{sample}" / "alignments_intersecting_mirna.sam",
             sample=pd.unique(samples_table.index.values),
         ),
-        intersect_sam=expand(
-            OUT_DIR / "{sample}" / "alignments_intersecting_mirna_sorted_tag.sam",
-            sample=pd.unique(samples_table.index.values),
-        ),
         table=expand(
             OUT_DIR / "TABLES" / "all_{mir}_counts.tab",
             mir=[mir for mir in config["mir_list"] if mir != "isomir"],
         ),
+        uncollapsed_sam=expand(
+            OUT_DIR / "{sample}" / "alignments_intersecting_mirna_uncollapsed.sam",
+            sample=pd.unique(samples_table.index.values),
+        ),
         uncollapsed_bam=expand(
             OUT_DIR
             / "{sample}"