NSFS | versioning | add comment and documentation about multipart upload version-id #8785
+10
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nadavMiz
force-pushed
the
multipart-upload-document
branch
from
88fa612 to
fa56b2d
Compare
romayalon
reviewed
Feb 13, 2025
…oad version-id Signed-off-by: nadav mizrahi <[email protected]>
nadavMiz
force-pushed
the
multipart-upload-document
branch
from
fa56b2d to
20c3d8e
Compare
shirady
reviewed
Feb 13, 2025
| @@ -49,6 +49,9 @@ In the figure, each original directory contains a hidden .versions/ sub- directo | |||
| * The version ID will be attached as an xattr of the file. | |||
| * When a latest version is moved to past versions directory, the file name will be changed from key to key_{version_id}. | |||
|
|
|||
| #### Multipart upload version order | |||
| according to AWS specifications multipart upload version time should be calcluated based on multipart upload creation time rather then completion time. For directory buckets, the object creation time is the completion date of the multipart upload. In our design the version-id time is calculated based on completion time. see https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-using-multipart-upload.html#s3-express-distributedmpupload | |||
There was a problem hiding this comment.
minor typos
I would move this part before the "out of scope" as it is quite an internal and deep subject (and you attached between basic concepts - version ID and Delete Marker).
I'm adding a suggestion for adding links and separation - accept only the parts that you like.
Suggested change
| according to AWS specifications multipart upload version time should be calcluated based on multipart upload creation time rather then completion time. For directory buckets, the object creation time is the completion date of the multipart upload. In our design the version-id time is calculated based on completion time. see https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-using-multipart-upload.html#s3-express-distributedmpupload | |
| According to AWS specifications, multipart upload version time should be calculated based on multipart upload creation time rather than completion time (see [AWS Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html#distributedmpupload)). | |
| On the other hand, for directory buckets, the object creation time is the completion date of the multipart upload (see [AWS Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-using-multipart-upload.html#s3-express-distributedmpupload). | |
| In our design, the version-id time is calculated based on completion time. |
| // 2.1.2. else - copy the prev part size file prefix to upload_path | ||
| // 3. copy bytes of the current's part size file | ||
| // NOTE on versioning - according to general aws specifications, the version_id time should be based on when we created the upload. | ||
| // for directory buckets, on AWS, the object creation time is the completion date of the multipart upload | ||
| // on our design we desided to do it based on when the upload was completed. |
There was a problem hiding this comment.
I would note the reason in short (1 to a couple of words - "performance"...).
Suggested change
| // on our design we desided to do it based on when the upload was completed. | |
| // on our design we decided to do it based on when the upload was completed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explain the changes
Issues: Fixed #xxx / Gap #xxx
Testing Instructions: