Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Back up adata fills up disk space #6

Open
eboileau opened this issue Jun 3, 2022 · 0 comments
Open

Back up adata fills up disk space #6

eboileau opened this issue Jun 3, 2022 · 0 comments

Comments

@eboileau
Copy link
Collaborator

eboileau commented Jun 3, 2022

In h5ad_identify_variable_genes.cgi, line 74 adata.filename = dest_datafile_path + ".backed.h5ad" writes adata as a dense matrix. There was this comment: This next command hits a memory issue if not backed. But the previous commands fail for various reasons if backed. So back up now.

The problem is that writing a dense matrix can take up to ~50GB or even more, even after filtering, and thus not only quickly fills diskspace under /tmp, but takes a lot of time to write... In all cases, in our current test setup, this ends up with No space left on device...

@eboileau eboileau mentioned this issue Jun 8, 2022
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant