Improve serialization efficiency when objects are proxied #120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Proxied objects end up being serialized twice: first when the object is serialized with Colmena to check the size of the serialized object and then again by ProxyStore when the object is proxied. ProxyStore supports passing custom serializer/deserializer functions so this commit adds a shim serializer and a deserializer wrapper to minimize serialization overheads when an object gets proxied. Note that there is still one extra step when using Colmena's pickle serialization method and ProxyStore. SerializationMethod.serialize will pickle the object producing a byte string then convert those bytes to a hex string then the shims convert that hex string back to bytes. I.e., there's this intermediate string representation that's created then discarded in this path. I don't really see a way of avoiding this because its an artifact of the different serialization types between Colmena and ProxyStore (str vs bytes).
|
LGTM. Thanks, @gpauloski ! |
gpauloski
added a commit
that referenced
this pull request
Feb 21, 2024
PR #120 introduced some shims for converting between Colmena's serialization utilities and ProxyStore's expected serialized type of bytes. One of these shims had a typo of "serialize" instead of "deserialize". I think this did not get caught by CI because there's no test for queues with ProxyStore enabled, there's just tests for manually passing proxies as function inputs.
gpauloski
added a commit
that referenced
this pull request
Feb 21, 2024
Fix proxy deserialization bug from #120
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To avoid double serialization of objects that end up being proxied, this PR passes a shim serializer and a deserializer wrapper to
store.proxy()to take advantage of the fact we already have a serialized string when proxying an object.Let me know what you think about this solution. I think it's the simplest in terms of implementation, but as I note in the commit message and code comment, there's still this conversion from
bytes -> str -> byteswhen using pickle as the Colmena serialization method.Also increases the max supported version of ProxyStore to allow v0.6.
Fixes #117