Stop downloading entire segments when retrieving objects #3362
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teor2345
added
bug
vedhavyas
reviewed
Jan 31, 2025
| @@ -13,7 +13,7 @@ use tracing::{debug, trace}; | |||
| // This code was copied and modified from subspace_service::sync_from_dsn::download_and_reconstruct_blocks(): | |||
| // <https://github.com/autonomys/subspace/blob/d71ca47e45e1b53cd2e472413caa23472a91cd74/crates/subspace-service/src/sync_from_dsn/import_blocks.rs#L236-L322> | |||
| pub async fn download_pieces<PG>( | |||
| piece_indexes: &Vec<PieceIndex>, | |||
| piece_indexes: &[PieceIndex], | |||
There was a problem hiding this comment.
if we want to clone anyway, wouldn't it be better to send the copy here instead ?
| pub fn decode_variant_and_length<I: Input>( | ||
| input: &mut I, | ||
| ) -> Result<(u8, Option<u64>), parity_scale_codec::Error> { | ||
| let variant = input |
There was a problem hiding this comment.
why not just call the decode_variant fn here?
| // All of the above, plus the partial byte count for the partial progress variant. | ||
| pub const MAX_SEGMENT_HEADER_SIZE: usize = MIN_SEGMENT_HEADER_SIZE + 4; | ||
|
|
||
| /// The maximum object length the implementation in this module can reliably handle. |
| /// Currently objects are limited by the largest block size in any domain, which is 5 MB. | ||
| /// But this implementation supports the maximum length of the 4 byte scale encoding. | ||
| pub const MAX_SUPPORTED_OBJECT_LENGTH: usize = 1024 * 1024 * 1024 - 1; | ||
| /// Currently objects are limited by the largest block size in any domain, which is 5 MB. But this |
There was a problem hiding this comment.
not sure if domain is the right word here since it we have a different context where we use it for domains
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes two performance edge cases in object retrieval:
Instead, it:
Other Changes
This PR fixes a miscalculation of segment offsets, which was never changed (or tested) since 2023, when the segment format was rewritten to remove the encoded number of segment items.
It also does some minor refactors, and adds some utility methods on pieces and segments.
TODO
Code contributor checklist: