Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a better ReliableStorageLock based on If-Match S3 atomic operations #2028

Open
IvoDD opened this issue Dec 2, 2024 · 2 comments
Open
Assignees
Labels
enhancement New feature or request replicated

Comments

@IvoDD
Copy link
Collaborator

IvoDD commented Dec 2, 2024

Is your feature request related to a problem? Please describe.
The current ReliableStorageLock was being implemented before If-Match on put objects was announced by aws and hence uses a more complex structure than it is needed. This ticket lists all minor things we need to address when doing this simplification.

This is blocked on aws-sdk-cpp not having 1.11.455 or later which has the new If-Match on put requests support.

Describe the solution you'd like

  • Use lock which just relies on a single ref key which gets overriden with If-Match statements.
  • Think about how existing uses of the lock can upgrade to the new locking
  • Rework C++ ReliableStorageLock tests to use a MockS3Client instead of the InMemoryStore. more info
  • Make the storage lock configure the underlying storage with a MaxTimeoutPoliocy. more info
  • Use Last-Modified time from S3 for expiration instead of the process time. more info
  • Move slower cpp stress tests to a separate target. more_info
@IvoDD IvoDD added the enhancement New feature or request label Dec 2, 2024
@IvoDD IvoDD self-assigned this Dec 2, 2024
@IvoDD IvoDD mentioned this issue Dec 2, 2024
5 tasks
@IvoDD
Copy link
Collaborator Author

IvoDD commented Dec 3, 2024

Also we need to think about:

  • Dynamically detecting whether storage supports atomic operations
  • Provide a means to fallback to the old storage lock (which will need some new APIs e.g. extending a lock)

@IvoDD
Copy link
Collaborator Author

IvoDD commented Dec 3, 2024

Rework C++ ReliableStorageLock tests to use a MockS3Client instead of the InMemoryStore. #2014 (comment)

Attempting to write such python tests turned out futile because of the GIL handling. It would probably still be better to have the C++ tests use MockS3Client instead of InMemoryStore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request replicated
Projects
None yet
Development

No branches or pull requests

2 participants