Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First release PR (version bumped to 1.0.0) #25

Merged
merged 173 commits into from
Feb 5, 2025
Merged
Show file tree
Hide file tree
Changes from 165 commits
Commits
Show all changes
173 commits
Select commit Hold shift + click to select a range
7a768c5
modules installed
vagkaratzas Nov 5, 2024
821f73e
input samplesheet update to fasta
vagkaratzas Nov 6, 2024
84f8965
clustering modules chained
vagkaratzas Nov 6, 2024
e5bc24f
nf-core createtsv test init
vagkaratzas Nov 6, 2024
1802206
debugged targetdb copy by staging
vagkaratzas Nov 6, 2024
40e02f2
syncing samples by meta, module diffs
vagkaratzas Nov 8, 2024
d362d9e
EXECUTE_CLUSTERING subworkflow done
vagkaratzas Nov 8, 2024
af91b89
Merge pull request #1 from vagkaratzas/createtsv-tests
vagkaratzas Nov 8, 2024
0c56473
chunk_clusters nf and py init
vagkaratzas Nov 11, 2024
ace2dbd
cluster chunkcs complete
vagkaratzas Nov 11, 2024
65381a7
Merge pull request #2 from vagkaratzas/chunking_clusters
vagkaratzas Nov 11, 2024
b1309f5
seqera container for local python module
vagkaratzas Nov 11, 2024
a84dbe6
clustering parameters added, chunking fasta instead of tsv to then fe…
vagkaratzas Nov 11, 2024
2e46012
removed pandas from chunking script, optimised for memory
vagkaratzas Nov 11, 2024
f2983bc
msa input formatted, famsa run properly
vagkaratzas Nov 12, 2024
6f52683
meta updated to chunks level, hmmbuild chained to workflow
vagkaratzas Nov 12, 2024
2d48faf
HMMER_HMMSEARCH chained, inputs combined successfully on meta id, met…
vagkaratzas Nov 12, 2024
54f7a05
minor publish path change
vagkaratzas Nov 12, 2024
5d9e7e9
hmmsearch evalue cutoff and output full MSA by default
vagkaratzas Nov 13, 2024
a39d649
generate_families converted into subworkflow
vagkaratzas Nov 13, 2024
e203af3
mafft alternative alignment tool
vagkaratzas Nov 13, 2024
e24a63b
hmmbuild updated
vagkaratzas Nov 14, 2024
4c639f0
added support for .gz input fasta
vagkaratzas Nov 14, 2024
b4a054f
Preparing input channel to extract family representative sequences fo…
vagkaratzas Nov 14, 2024
2b6b4a8
added clipkit module, fixing some linting warnings, updated subworkflows
vagkaratzas Nov 14, 2024
c136435
extract family reps script
vagkaratzas Nov 19, 2024
5e068d1
removed prokka module
vagkaratzas Nov 19, 2024
b0ee92d
export fam to rep mapping
vagkaratzas Nov 21, 2024
82742e7
EXTRACT_FAMILY_REPS publishDir
vagkaratzas Nov 21, 2024
8b444df
Merge pull request #4 from vagkaratzas/prokka
vagkaratzas Nov 21, 2024
ea8e300
custom multiqc with family metadata
vagkaratzas Nov 22, 2024
dfaae41
added label and when blocks to custom modules to fix linting warnings
vagkaratzas Nov 22, 2024
53ea42f
input samplesheet and test data hosted in nf-core test-datasets repo,…
vagkaratzas Nov 22, 2024
eb08a3c
updated mafft module to mafft-align
vagkaratzas Dec 5, 2024
5283390
subworkflows updated
vagkaratzas Dec 5, 2024
18be6da
updated readme and citations doc
vagkaratzas Dec 5, 2024
f7bc3f5
usage.md updated
vagkaratzas Dec 5, 2024
fb54976
clustering outputs updated and written in output.md
vagkaratzas Dec 5, 2024
c9ac121
aligners outputs updated
vagkaratzas Dec 5, 2024
1aa92cd
hmmer outputs updated + documentation
vagkaratzas Dec 5, 2024
619f701
gap_threshold aprameter for clipkit
vagkaratzas Dec 6, 2024
39a037f
mmseqs suite version bump
vagkaratzas Dec 6, 2024
e95a1ff
clip_ends only local module
vagkaratzas Dec 6, 2024
cbd1292
Merge pull request #5 from vagkaratzas/clip-ends-only
vagkaratzas Dec 6, 2024
86a950b
nf-core subworkflows update
vagkaratzas Dec 8, 2024
2df0b34
hmmsearch results filtering by length, fasta instead of stockholm ref…
vagkaratzas Dec 9, 2024
1a2d333
init redundancy_check subworkflow
vagkaratzas Dec 9, 2024
98fb26c
execute_clustering subworkflow updated and reused in redundancy remov…
vagkaratzas Dec 10, 2024
8ae6410
TODOs added to continue with redundancy mechanisms
vagkaratzas Dec 10, 2024
b42db21
remove_redundant_seqs module done, sequence align converted to subwor…
vagkaratzas Dec 11, 2024
7aa1e3c
concat_hmms module
vagkaratzas Dec 11, 2024
21c3d19
CONCAT_HMMS module, HMMER_HMMSEARCH for family redundancy checking, R…
vagkaratzas Dec 11, 2024
39afbb9
remove_self_hits function
vagkaratzas Dec 11, 2024
5ddbf77
filter_by_length
vagkaratzas Dec 11, 2024
69b054e
remove_redundant_fams module done and container updated to pandas
vagkaratzas Dec 11, 2024
480bc2a
filtered sequence names updated, results folder architecture updated
vagkaratzas Dec 12, 2024
e5e28dc
Merge pull request #6 from vagkaratzas/remove-family-redundancy
vagkaratzas Dec 12, 2024
178ff6b
Merge pull request #7 from vagkaratzas/redundancy_check
vagkaratzas Dec 12, 2024
2f0e75c
readme and output.md doc updates
vagkaratzas Dec 13, 2024
7b16c7c
metro map updated
vagkaratzas Dec 13, 2024
dfe39fb
non redundant hmm filtering
vagkaratzas Dec 16, 2024
41ffa12
Template update for nf-core/tools version 3.1.0
vagkaratzas Dec 16, 2024
f9ea509
Template update for nf-core/tools version 3.1.0
vagkaratzas Dec 16, 2024
940659c
Merge branch 'sync' into merge_template
vagkaratzas Dec 16, 2024
9f22b4e
Merge pull request #8 from vagkaratzas/merge_template
vagkaratzas Dec 16, 2024
92a3fdb
manifest update and modules config proper sourcing
vagkaratzas Dec 16, 2024
8e6d4bd
Merge pull request #9 from vagkaratzas/sync
vagkaratzas Dec 16, 2024
baf48e9
future proofing multiqc numeric protein name bug (proper strings will…
vagkaratzas Dec 16, 2024
97fcd22
smallest test skipping clipping of msas and redundancy removal mechanism
vagkaratzas Dec 16, 2024
ca3b434
make recruiting of sequences with hmmsearch optional, both tests runn…
vagkaratzas Dec 16, 2024
f64d5de
Merge pull request #11 from vagkaratzas/bug-fix--multiqc-numeric-prot…
vagkaratzas Dec 17, 2024
0f2bda3
minor reformats
vagkaratzas Dec 17, 2024
7835d65
Merge pull request #10 from vagkaratzas/new-test--minimal-test-withou…
vagkaratzas Dec 17, 2024
ec73bfa
This changeset included a WIP update workflow.
mberacochea Dec 20, 2024
6d1083b
Fix a missing bit in the update
mberacochea Dec 22, 2024
5ab2ce6
This changeset included a WIP update workflow.
mberacochea Dec 20, 2024
42a6105
Fix a missing bit in the update
mberacochea Dec 22, 2024
deb184b
Merge branch 'update_subworkflow_vag' of https://github.com/vagkaratz…
vagkaratzas Dec 23, 2024
185802e
correct and rearrange stuff in remove_redundancy subworkflow
vagkaratzas Dec 23, 2024
1cb45ff
sync to latest template
vagkaratzas Dec 23, 2024
8b36bd3
nf-core yml version updated
vagkaratzas Dec 23, 2024
0a6b233
multiQC stuff undo linting
vagkaratzas Dec 23, 2024
8437b96
update_families moved to its own subworkflow
vagkaratzas Dec 23, 2024
2dd40f7
restructured update_families until hmmer_hmmsearch
vagkaratzas Dec 23, 2024
c7c3df6
branch hits fasta
vagkaratzas Dec 24, 2024
71c39a1
cat_cat module config
vagkaratzas Dec 24, 2024
7d0ee54
removed unused modules, binding proper family msas with recruited seq…
vagkaratzas Dec 24, 2024
c2e20f1
seqkit + cat fasta with family specific msa
vagkaratzas Dec 24, 2024
7f1a7c6
prefixes up to clustering updated
vagkaratzas Dec 24, 2024
ae129a2
update_families subworkflow done
vagkaratzas Dec 24, 2024
7aa8c0b
clipends file extension updated
vagkaratzas Dec 24, 2024
a723a42
metro map and output.md updated
vagkaratzas Dec 25, 2024
3d17c4f
updated readme and citations
vagkaratzas Dec 25, 2024
59b006a
validation schema updated to allow either existing hmm and msa folder…
vagkaratzas Dec 26, 2024
b71c210
untar logic implemented, removed support for simple folders due to co…
vagkaratzas Dec 26, 2024
e31cdc9
test configs updated
vagkaratzas Dec 26, 2024
6c9008c
samplesheet description updated
vagkaratzas Dec 26, 2024
da80843
Merge pull request #14 from vagkaratzas/test_data_update
vagkaratzas Jan 2, 2025
01502b7
Update README.md
vagkaratzas Jan 2, 2025
27de55f
python errors to stderr
vagkaratzas Jan 2, 2025
e81cc77
Update workflows/proteinfamilies.nf
vagkaratzas Jan 2, 2025
2ebf65a
Update docs/output.md
vagkaratzas Jan 2, 2025
37a2c64
Update docs/output.md
vagkaratzas Jan 2, 2025
3406075
Update docs/output.md
vagkaratzas Jan 2, 2025
5d93651
syncing of seqs and clusters after clustering done inside the execute…
vagkaratzas Jan 2, 2025
99a62a9
file.getSimpleName
vagkaratzas Jan 2, 2025
b542eb6
chunk in REMOVE_REDUNDANT_SEQS tag
vagkaratzas Jan 2, 2025
b98c514
Update docs/output.md
vagkaratzas Jan 2, 2025
c0cbae9
regex for parsed hit sequence name and range
vagkaratzas Jan 3, 2025
81d0d67
validateMatchingFolders groovy function to check for valid existing H…
vagkaratzas Jan 3, 2025
bb82f71
Merge pull request #13 from vagkaratzas/update_subworkflow_vag
vagkaratzas Jan 7, 2025
de8421f
branch hits fasta updated to use env coords of domtbl
vagkaratzas Jan 9, 2025
ed1cc98
tryCatch block added in filter_recruited function of branch_hits_fasta
vagkaratzas Jan 9, 2025
040464e
filter recruited updated -not using auto full alignment from hmmsearc…
vagkaratzas Jan 10, 2025
7577bcc
HMMER_HMMALIGN added, checkpoint
vagkaratzas Jan 10, 2025
85272bd
remove redundancy updated to work with sto alignments as well
vagkaratzas Jan 10, 2025
7a74d6e
metro map updated with hmmalign
vagkaratzas Jan 10, 2025
49acd97
metro updated with hmmalign
vagkaratzas Jan 10, 2025
54980f7
output.md updated
vagkaratzas Jan 10, 2025
51e0b9b
filter_non_redundant_hmms updated to work properly even without fishi…
vagkaratzas Jan 10, 2025
bf099ff
Merge pull request #1 from vagkaratzas/using-envelope-coords
vagkaratzas Jan 10, 2025
dc80ef6
Merge pull request #15 from vagkaratzas/dev
vagkaratzas Jan 10, 2025
f4849e9
usage.md samplesheet input description update
vagkaratzas Jan 10, 2025
9f022f3
Merge pull request #18 from vagkaratzas/dev
vagkaratzas Jan 10, 2025
22d01d2
Template update for nf-core/tools version 3.1.2
nf-core-bot Jan 20, 2025
36bc897
Merge branch 'dev' into nf-core-template-merge-3.1.2
vagkaratzas Jan 27, 2025
6d4442c
Merge pull request #19 from nf-core/nf-core-template-merge-3.1.2
vagkaratzas Jan 27, 2025
dc5a884
modules update (mmseqs, multiqc, seqkit)
vagkaratzas Jan 27, 2025
f630ac0
Merge pull request #20 from vagkaratzas/dev
vagkaratzas Jan 27, 2025
77db71c
Template update for nf-core/tools version 3.2.0
nf-core-bot Jan 27, 2025
3b377ab
Update README.md
vagkaratzas Jan 28, 2025
251f7e7
Merge branch 'dev' into nf-core-template-merge-3.2.0
vagkaratzas Jan 28, 2025
f3caa75
Merge pull request #21 from nf-core/nf-core-template-merge-3.2.0
vagkaratzas Jan 28, 2025
6b877be
local modules mem reqs in base.config
vagkaratzas Jan 28, 2025
906eec2
Merge pull request #22 from vagkaratzas/dev
vagkaratzas Jan 28, 2025
704ebe4
metro map updated and svg added
vagkaratzas Jan 31, 2025
a5a8383
changelog updated for first release
vagkaratzas Jan 31, 2025
02ec8aa
Biopython citation added
vagkaratzas Jan 31, 2025
891706b
schema pattern updated to include faa and fas input files
vagkaratzas Jan 31, 2025
228659e
Update assets/schema_input.json
vagkaratzas Jan 31, 2025
a777d52
gzipped tarball description specification
vagkaratzas Jan 31, 2025
e83d6b4
python script licensing
vagkaratzas Jan 31, 2025
00c2c6f
Update conf/test_full.config
vagkaratzas Jan 31, 2025
e3db3aa
resourceLimits removed from test_full
vagkaratzas Jan 31, 2025
bdb1bec
removed unused process block from test_full config
vagkaratzas Jan 31, 2025
bc594f8
Parameter specifications in usage.md
vagkaratzas Jan 31, 2025
dd2cc97
Update subworkflows/local/utils_nfcore_proteinfamilies_pipeline/main.nf
vagkaratzas Jan 31, 2025
07d52a3
checkIfExists: true added to file()
vagkaratzas Jan 31, 2025
f6cb471
Update nextflow_schema.json
vagkaratzas Jan 31, 2025
0ce6ee0
Update nextflow.config
vagkaratzas Jan 31, 2025
f050d84
Update docs/output.md
vagkaratzas Jan 31, 2025
1b2d81b
Update docs/usage.md
vagkaratzas Jan 31, 2025
921cf06
Update docs/output.md
vagkaratzas Jan 31, 2025
bdc8627
batch commit suggestions from james
vagkaratzas Jan 31, 2025
0ab063f
ch_ notation instead of set {}, .map changed to multiline everywhere,…
vagkaratzas Jan 31, 2025
9be00f7
assets/samplesheet.csv added back
vagkaratzas Jan 31, 2025
e39a349
modules and subworkflows moved to respective folders + env
vagkaratzas Jan 31, 2025
f64c769
prefix instead of meta.id for output naming in two of the local modules
vagkaratzas Jan 31, 2025
d6e2e3e
only main author mail kept
vagkaratzas Jan 31, 2025
86cdb61
removed duplicate includeConfig 'conf/modules.config'
vagkaratzas Jan 31, 2025
99de435
interpretation of folder outputs in output.md, strict clustering thre…
vagkaratzas Jan 31, 2025
ada9368
help_text in most schema params
vagkaratzas Jan 31, 2025
acb9465
Merge pull request #23 from vagkaratzas/dev
vagkaratzas Jan 31, 2025
d3a9e21
first release version bump (1.0.0)
vagkaratzas Feb 3, 2025
16c1ca9
Merge pull request #24 from vagkaratzas/dev
vagkaratzas Feb 3, 2025
6ffb2df
Update CHANGELOG.md
jfy133 Feb 5, 2025
a380e49
[automated] Fix code linting
nf-core-bot Feb 5, 2025
506c005
Update docs/output.md
vagkaratzas Feb 5, 2025
cf86243
Update subworkflows/local/utils_nfcore_proteinfamilies_pipeline/main.nf
vagkaratzas Feb 5, 2025
528d493
minor typo fix
vagkaratzas Feb 5, 2025
f2dab99
Merge pull request #26 from vagkaratzas/dev
vagkaratzas Feb 5, 2025
dbf534c
folder.toUriString()
vagkaratzas Feb 5, 2025
64e43ca
Merge pull request #27 from vagkaratzas/dev
vagkaratzas Feb 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,7 @@ indent_size = unset
# ignore python and markdown
[*.{py,md}]
indent_style = unset

# ignore ro-crate metadata files
[**/ro-crate-metadata.json]
insert_final_newline = unset
12 changes: 6 additions & 6 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# nf-core/proteinfamilies: Contributing Guidelines
# `nf-core/proteinfamilies`: Contributing Guidelines

Hi there!
Many thanks for taking an interest in improving nf-core/proteinfamilies.
Expand Down Expand Up @@ -55,23 +55,23 @@ These tests are run both with the latest available version of `Nextflow` and als

:warning: Only in the unlikely and regretful event of a release happening with a bug.

- On your own fork, make a new branch `patch` based on `upstream/master`.
- On your own fork, make a new branch `patch` based on `upstream/main` or `upstream/master`.
- Fix the bug, and bump version (X.Y.Z+1).
- A PR should be made on `master` from patch to directly this particular bug.
- Open a pull-request from `patch` to `main`/`master` with the changes.

## Getting help

For further information/help, please consult the [nf-core/proteinfamilies documentation](https://nf-co.re/proteinfamilies/usage) and don't hesitate to get in touch on the nf-core Slack [#proteinfamilies](https://nfcore.slack.com/channels/proteinfamilies) channel ([join our Slack here](https://nf-co.re/join/slack)).

## Pipeline contribution conventions

To make the nf-core/proteinfamilies code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.
To make the `nf-core/proteinfamilies` code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.

### Adding a new step

If you wish to contribute a new step, please use the following coding standards:

1. Define the corresponding input channel into your new process from the expected previous process channel
1. Define the corresponding input channel into your new process from the expected previous process channel.
2. Write the process block (see below).
3. Define the output channel if needed (see below).
4. Add any new parameters to `nextflow.config` with a default (see below).
Expand All @@ -84,7 +84,7 @@ If you wish to contribute a new step, please use the following coding standards:

### Default values

Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
Parameters should be initialised / defined with default values within the `params` scope in `nextflow.config`.

Once there, use `nf-core pipelines schema build` to add to `nextflow_schema.json`.

Expand Down
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ body:

- [nf-core website: troubleshooting](https://nf-co.re/usage/troubleshooting)
- [nf-core/proteinfamilies pipeline documentation](https://nf-co.re/proteinfamilies/usage)

- type: textarea
id: description
attributes:
Expand Down
21 changes: 17 additions & 4 deletions .github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
name: nf-core AWS full size tests
# This workflow is triggered on PRs opened against the master branch.
# This workflow is triggered on PRs opened against the main/master branch.
# It can be additionally triggered manually with GitHub actions workflow dispatch button.
# It runs the -profile 'test_full' on AWS batch

on:
pull_request:
branches:
- main
- master
workflow_dispatch:
pull_request_review:
Expand All @@ -18,18 +19,30 @@ jobs:
if: github.repository == 'nf-core/proteinfamilies' && github.event.review.state == 'approved' && github.event.pull_request.base.ref == 'master' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: octokit/[email protected]
- name: Get PR reviews
uses: octokit/[email protected]
if: github.event_name != 'workflow_dispatch'
id: check_approvals
continue-on-error: true
with:
route: GET /repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews
route: GET /repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews?per_page=100
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- id: test_variables

- name: Check for approvals
if: ${{ failure() && github.event_name != 'workflow_dispatch' }}
run: |
echo "No review approvals found. At least 2 approvals are required to run this action automatically."
exit 1

- name: Check for enough approvals (>=2)
id: test_variables
if: github.event_name != 'workflow_dispatch'
run: |
JSON_RESPONSE='${{ steps.check_approvals.outputs.data }}'
CURRENT_APPROVALS_COUNT=$(echo $JSON_RESPONSE | jq -c '[.[] | select(.state | contains("APPROVED")) ] | length')
test $CURRENT_APPROVALS_COUNT -ge 2 || exit 1 # At least 2 approvals are required

- name: Launch workflow via Seqera Platform
uses: seqeralabs/action-tower-launch@v2
# TODO nf-core: You can customise AWS full pipeline tests as required
Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/branch.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
name: nf-core branch protection
# This workflow is triggered on PRs to master branch on the repository
# It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev`
# This workflow is triggered on PRs to `main`/`master` branch on the repository
# It fails when someone tries to make a PR against the nf-core `main`/`master` branch instead of `dev`
on:
pull_request_target:
branches: [master]
branches:
- main
- master

jobs:
test:
runs-on: ubuntu-latest
steps:
# PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches
# PRs to the nf-core repo main/master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches
- name: Check PRs
if: github.repository == 'nf-core/proteinfamilies'
run: |
Expand All @@ -22,7 +24,7 @@ jobs:
uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
with:
message: |
## This PR is against the `master` branch :x:
## This PR is against the `${{github.event.pull_request.base.ref}}` branch :x:

* Do not close this PR
* Click _Edit_ and change the `base` to `dev`
Expand All @@ -32,9 +34,9 @@ jobs:

Hi @${{ github.event.pull_request.user.login }},

It looks like this pull-request is has been made against the [${{github.event.pull_request.head.repo.full_name }}](https://github.com/${{github.event.pull_request.head.repo.full_name }}) `master` branch.
The `master` branch on nf-core repositories should always contain code from the latest release.
Because of this, PRs to `master` are only allowed if they come from the [${{github.event.pull_request.head.repo.full_name }}](https://github.com/${{github.event.pull_request.head.repo.full_name }}) `dev` branch.
It looks like this pull-request is has been made against the [${{github.event.pull_request.head.repo.full_name }}](https://github.com/${{github.event.pull_request.head.repo.full_name }}) ${{github.event.pull_request.base.ref}} branch.
The ${{github.event.pull_request.base.ref}} branch on nf-core repositories should always contain code from the latest release.
Because of this, PRs to ${{github.event.pull_request.base.ref}} are only allowed if they come from the [${{github.event.pull_request.head.repo.full_name }}](https://github.com/${{github.event.pull_request.head.repo.full_name }}) `dev` branch.

You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
Note that even after this, the test will continue to show as failing until you push a new commit.
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ jobs:
profile: "singularity"
steps:
- name: Check out pipeline code
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
fetch-depth: 0

- name: Set up Nextflow
uses: nf-core/setup-nextflow@v2
Expand Down
59 changes: 37 additions & 22 deletions .github/workflows/download_pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Test successful pipeline download with 'nf-core pipelines download'

# Run the workflow when:
# - dispatched manually
# - when a PR is opened or reopened to master branch
# - when a PR is opened or reopened to main/master branch
# - the head branch of the pull request is updated, i.e. if fixes for a release are pushed last minute to dev.
on:
workflow_dispatch:
Expand All @@ -17,25 +17,42 @@ on:
- edited
- synchronize
branches:
- main
- master
pull_request_target:
branches:
- main
- master

env:
NXF_ANSI_LOG: false

jobs:
configure:
runs-on: ubuntu-latest
outputs:
REPO_LOWERCASE: ${{ steps.get_repo_properties.outputs.REPO_LOWERCASE }}
REPOTITLE_LOWERCASE: ${{ steps.get_repo_properties.outputs.REPOTITLE_LOWERCASE }}
REPO_BRANCH: ${{ steps.get_repo_properties.outputs.REPO_BRANCH }}
steps:
- name: Get the repository name and current branch
id: get_repo_properties
run: |
echo "REPO_LOWERCASE=${GITHUB_REPOSITORY,,}" >> "$GITHUB_OUTPUT"
echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> "$GITHUB_OUTPUT"
echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> "$GITHUB_OUTPUT"

download:
runs-on: ubuntu-latest
needs: configure
steps:
- name: Install Nextflow
uses: nf-core/setup-nextflow@v2

- name: Disk space cleanup
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1

- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.12"
architecture: "x64"
Expand All @@ -50,12 +67,6 @@ jobs:
python -m pip install --upgrade pip
pip install git+https://github.com/nf-core/tools.git@dev

- name: Get the repository name and current branch set as environment variable
run: |
echo "REPO_LOWERCASE=${GITHUB_REPOSITORY,,}" >> ${GITHUB_ENV}
echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> ${GITHUB_ENV}
echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> ${GITHUB_ENV}

- name: Make a cache directory for the container images
run: |
mkdir -p ./singularity_container_images
Expand All @@ -64,55 +75,59 @@ jobs:
env:
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
run: |
nf-core pipelines download ${{ env.REPO_LOWERCASE }} \
--revision ${{ env.REPO_BRANCH }} \
--outdir ./${{ env.REPOTITLE_LOWERCASE }} \
nf-core pipelines download ${{ needs.configure.outputs.REPO_LOWERCASE }} \
--revision ${{ needs.configure.outputs.REPO_BRANCH }} \
--outdir ./${{ needs.configure.outputs.REPOTITLE_LOWERCASE }} \
--compress "none" \
--container-system 'singularity' \
--container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io" \
--container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io/library/" \
--container-cache-utilisation 'amend' \
--download-configuration 'yes'

- name: Inspect download
run: tree ./${{ env.REPOTITLE_LOWERCASE }}
run: tree ./${{ needs.configure.outputs.REPOTITLE_LOWERCASE }}

- name: Inspect container images
run: tree ./singularity_container_images | tee ./container_initial

- name: Count the downloaded number of container images
id: count_initial
run: |
image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
echo "Initial container image count: $image_count"
echo "IMAGE_COUNT_INITIAL=$image_count" >> ${GITHUB_ENV}
echo "IMAGE_COUNT_INITIAL=$image_count" >> "$GITHUB_OUTPUT"

- name: Run the downloaded pipeline (stub)
id: stub_run_pipeline
continue-on-error: true
env:
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
NXF_SINGULARITY_HOME_MOUNT: true
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -stub -profile test,singularity --outdir ./results
run: nextflow run ./${{needs.configure.outputs.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ needs.configure.outputs.REPO_BRANCH }}) -stub -profile test,singularity --outdir ./results
- name: Run the downloaded pipeline (stub run not supported)
id: run_pipeline
if: ${{ job.steps.stub_run_pipeline.status == failure() }}
if: ${{ steps.stub_run_pipeline.outcome == 'failure' }}
env:
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
NXF_SINGULARITY_HOME_MOUNT: true
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -profile test,singularity --outdir ./results
run: nextflow run ./${{ needs.configure.outputs.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ needs.configure.outputs.REPO_BRANCH }}) -profile test,singularity --outdir ./results

- name: Count the downloaded number of container images
id: count_afterwards
run: |
image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
echo "Post-pipeline run container image count: $image_count"
echo "IMAGE_COUNT_AFTER=$image_count" >> ${GITHUB_ENV}
echo "IMAGE_COUNT_AFTER=$image_count" >> "$GITHUB_OUTPUT"

- name: Compare container image counts
run: |
if [ "${{ env.IMAGE_COUNT_INITIAL }}" -ne "${{ env.IMAGE_COUNT_AFTER }}" ]; then
initial_count=${{ env.IMAGE_COUNT_INITIAL }}
final_count=${{ env.IMAGE_COUNT_AFTER }}
if [ "${{ steps.count_initial.outputs.IMAGE_COUNT_INITIAL }}" -ne "${{ steps.count_afterwards.outputs.IMAGE_COUNT_AFTER }}" ]; then
initial_count=${{ steps.count_initial.outputs.IMAGE_COUNT_INITIAL }}
final_count=${{ steps.count_afterwards.outputs.IMAGE_COUNT_AFTER }}
difference=$((final_count - initial_count))
echo "$difference additional container images were \n downloaded at runtime . The pipeline has no support for offline runs!"
tree ./singularity_container_images
tree ./singularity_container_images > ./container_afterwards
diff ./container_initial ./container_afterwards
exit 1
else
echo "The pipeline can be downloaded successfully!"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/fix-linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
steps:
# Use the @nf-core-bot token to check out so we can push later
- uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
token: ${{ secrets.nf_core_bot_auth_token }}

Expand All @@ -32,7 +32,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.nf_core_bot_auth_token }}

# Install and run pre-commit
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.12"

Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Set up Python 3.12
uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.12"

Expand All @@ -31,12 +31,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out pipeline code
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- name: Install Nextflow
uses: nf-core/setup-nextflow@v2

- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.12"
architecture: "x64"
Expand Down Expand Up @@ -74,7 +74,7 @@ jobs:

- name: Upload linting log file artifact
if: ${{ always() }}
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882 # v4
with:
name: linting-logs
path: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/linting_comment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Download lint results
uses: dawidd6/action-download-artifact@bf251b5aa9c2f7eeb574a96ee720e24f801b7c11 # v6
uses: dawidd6/action-download-artifact@20319c5641d495c8a52e688b7dc5fada6c3a9fbc # v8
with:
workflow: linting.yml
workflow_conclusion: completed
Expand Down
Loading
Loading