Hot archive #200
Merged
Hot archive #200
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 tasks
dmkozh
reviewed
Jul 30, 2024
dmkozh
approved these changes
Aug 8, 2024
SirTyson
added a commit
to SirTyson/stellar-xdr
that referenced
this pull request
Aug 8, 2024
dmkozh
pushed a commit
that referenced
this pull request
Aug 8, 2024
github-merge-queue bot
pushed a commit
to stellar/stellar-core
that referenced
this pull request
Nov 23, 2024
# Description Resolves #4394 Depends on XDR changes in stellar/stellar-xdr#200. This change refactors the BucketList so that we can have multiple Bucket Lists simultaneously. This is necessary for State Archival, where validators will maintain a live BucketList, hot archive BucketList, and cold archive BucketList. This change does not yet support writing the new archival BucketLists to history, Each BucketList has small differences wrt to the entry types it stores, merge logic, and how lookups are done, but the overall BucketList level logic is the same. Because of this, it seemed easiest to template the Bucket related classes and specialize a few functions. The difference are as follows (I'll be updating the Archival CAP with this info soon) Entry Types: Live BucketList: `BucketEntry` Hot Archive BucketList: `HotArchiveBucketEntry`. This change was necessary due to how LedgerKeys are stored. In the Live BucketList, only `DEADENTRY` store plain `LedgerKey`. `DEADENTRY` is equivalent to null, so these LedgerKeys are dropped at the bottom level bucket. Hot Archives require two types that store LedgerKeys: `HA_LIVE` and `HA_DELETED`. `HA_LIVE` is equivalent to the "null" type in the Live BucketList, and is dropped in the bottom bucket. However, `HA_DELETED` needs to be persisted. Merging: In the hot archive, `LedgerEntry` are never updated. They may be overwritten by another `HotArchiveBucketEntry` type, but the `LedgerEntry` contents itself never change. This means that the `INITENTRY`, `LIVEENTRY`, and `DEADENTRY` abstraction doesn't really make sense. This makes the merge logic for Hot Archive buckets very simple, the newer entry always overwrites the older entry. This PR adds the concept of a "Tombstone" entry. A tombstone entry essentially represents null in the given BucketList and can be dropped once it reaches the bottom bucket. On the live BucketList, DEADENTRY is the tombstone type. On the Hot Archive BucketList, `HA_LIVE` is the tombstone. BucketListDB lookup: The live BucketList only ever returns `LedgerEntry` types when queried. Any DEADENTRY LedgerKey is simply omitted from the return, as this is the "null" tombstone type. Hot archive lookups must return both LedgerKey and LedgerEntry types. HA_LIVE entries are of type LedgerKey and can be omitted from the return since they are the tombstone type. However, HA_ARCHIVED entries are not tombstone entries and must be returned when found. Do to this, Hot Archive lookups return `HotArchiveBucketEntry` instead of `LedgerEntry` when queried. Future updates will add more functionality to Hot Archives, further distinguishing it from the live BucketList, and add a third BucketList type called the cold archive. This cold archive will add a third BucketEntry type and have different merge logic as well, so I think the templated Bucket classes, while verbose, are warranted. This is currently a draft until the XDR changes are merged. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds
Hot Archivetypes to vNext.