Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default shared memory limits too restrictive #65

Open
mdales opened this issue Jul 23, 2024 · 3 comments
Open

Default shared memory limits too restrictive #65

mdales opened this issue Jul 23, 2024 · 3 comments

Comments

@mdales
Copy link
Collaborator

mdales commented Jul 23, 2024

I've hit this with docker (explained to me by @avsm) that the default limits on shared memory are quite restrictive in terms of the datasets we work with. I've been chasing some random sigbus errors in shark (process exits with signal 135) and I believe this is the same root cause.

This ticket is to remind me to look at making this configurable or at least have run use a higher bound. For now I can work around it by disabling parallelism in yirgacheffe

@patricoferris
Copy link
Collaborator

Potentially relevant is where we set this in the obuilder (runc) world -- currently to 65536k which is probably the same as docker: https://github.com/quantifyearth/obuilder/blob/850bd8bfd35370b6bdea4de9058804aed3c8b4e6/lib/sandbox.runc.ml#L202-L211

@mdales
Copy link
Collaborator Author

mdales commented Jul 26, 2024

Yeah, I believe docker is the same, but 64MB is too small in a geospatial context. I added parallel_save to which shares data via shared memory, and really I think in this context we should set it to a large %age of actual memory. But if Obuilder is merged we can easily make that change for our use case.

@mdales
Copy link
Collaborator Author

mdales commented Aug 2, 2024

Created this PR to raise it to 64GB for now. quantifyearth/obuilder#2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants