[resharding] Handling forks in memtrie during resharding #12714
+196
−80
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is the implementation for handling forks in memtrie which may occur close to or at the resharding boundary.
For context on what the exact problem is and the solution, please see this Zulip post.
High level, we want to achieve the following: In case of forks close to the resharding boundary
Then we want the process the fork block on top of both the children shards as well.
Main changes in this PR
temp_split_shard_map
that keeps track of parent memtrie splits. Note that we append "temp" in the name as we only keep track of splits where the child shares the frozen trie with parent. This entry should be removed as soon as the child memtrie becomes independent of parent.freeze_parent_memtrie
function, we updatetemp_split_shard_map
Trie
object now haschildren_memtries
member that stores reference to the children memtries. This is generated usingtemp_split_shard_map
from ShardTries.children_memtrie_changes
that saves the changes from parent that are to be written to children.update
function is modified to write the changes to children memtries along with parent memtrie. This is better viewed with "hide whitespaces" option.The resharding tests for forks still continue to fail as now we are hitting an issue with flat storage resharding. That is out the scope for this PR.