You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment when distributed memh5 datasets are written to disk, they are done so serially, with each rank waiting for its turn to write. This is clearly a bit dumb when running on nice parallel filesystems like GPFS (nice might be a bit generous).
The way to work around this is to:
Use one rank to create and pre-allocate the dataset as a contiguous dataset, also probably a good time to write out the attributes.
Close the file.
Redistribute the data to the slowest varying axis.
Have each rank open the file, figure out the offset into the file for its chunk of data and then lock the range of the data it needs.
Each rank writes its data and closes the file in parallel.
The text was updated successfully, but these errors were encountered:
At the moment when distributed
memh5
datasets are written to disk, they are done so serially, with each rank waiting for its turn to write. This is clearly a bit dumb when running on nice parallel filesystems like GPFS (nice might be a bit generous).The way to work around this is to:
The text was updated successfully, but these errors were encountered: