Skip to content

Commit

Permalink
torch: using memoryview for slicing in checkpoints
Browse files Browse the repository at this point in the history
Features: DfuseFind
Allow-unstable-test: true

Signed-off-by: Denis Barakhtanov <[email protected]>
  • Loading branch information
0xE0F committed Jan 28, 2025
1 parent 014f11e commit 8aab5b5
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions src/client/pydaos/torch/torch_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,8 @@ def write(self, data):
self._position += len(data)
return len(data)

# Creating memoryview to avoid copying the data on chunking

Check warning on line 329 in src/client/pydaos/torch/torch_api.py

View workflow job for this annotation

GitHub Actions / Pylint check

wrong-spelling-in-comment, Wrong spelling of a word 'memoryview' in a comment:
data = memoryview(data)
written = len(data)
while len(data) > 0:
fit = min(len(data), self._transfer_chunk_size - len(self._buffer))
Expand Down Expand Up @@ -380,8 +382,8 @@ def _flush(self):
def _submit_chunk(self, offset, chunk):
""" Submits chunk for writing to the container.
It will block if the queue is full and has a size limit, forcing the caller to wait
until some of the chunks are written to the storage.
This is blocking methind, if the queue is bounded (via chunks_limit parameter) and full,

Check warning on line 385 in src/client/pydaos/torch/torch_api.py

View workflow job for this annotation

GitHub Actions / Pylint check

wrong-spelling-in-docstring, Wrong spelling of a word 'methind' in a docstring:
forcing the caller to wait until some of the chunks are written to the storage.
"""

self._queue.put((offset, chunk))
Expand Down

0 comments on commit 8aab5b5

Please sign in to comment.