Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No error on GPU OOM Colabfold #245

Open
ppenev-eligo opened this issue Feb 6, 2025 · 0 comments
Open

No error on GPU OOM Colabfold #245

ppenev-eligo opened this issue Feb 6, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@ppenev-eligo
Copy link

ppenev-eligo commented Feb 6, 2025

Description of the bug

When submitting the pipeline through Seqera the platform does not evoke a 1 exit status if colabfold runs out of memory on GPU. The pipeline returns a success 0 status which prevents retries with different GPUs.

Command used and terminal output

Generated through Seqera:

nextflow run 'https://github.com/nf-core/proteinfold' -name test_run -params-file 'https://api.cloud.seqera.io/ephemeral/[id].json' -with-tower -r [tower-hash]

Relevant files

task-6.command.out.txt

task-6.command.err.txt

System information

  • Nextflow v24.10.3
  • Google Cloud
  • Executor = 'google-batch'
  • container Docker [COLABFOLD_BATCH:nf-core/proteinfold_colabfold:dev]
  • OS Linux DeepLearning VM with CUDA preinstalled; However the CUDA on the docker image is 12.6 which solves some jax issues.
  • nf-core/proteinfold v1.1.1
@ppenev-eligo ppenev-eligo added the bug Something isn't working label Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant