Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimistic block pool #12788

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Conversation

Longarithm
Copy link
Member

@Longarithm Longarithm commented Jan 24, 2025

Introduce the pool for storing OptimisticBlocks, joining them with chunks, executing them and reusing cached results. #10584

The change is pretty big; however, I think it's important to merge at once because it already gives the working example. I'll describe 2 major changes which are sufficient to review.

OptimisticBlockChunksPool

Receives optimistic block (OB), currently from block producer itself only. Receives chunks from ShardsManager. When, on top of some prev block, OB and chunks are received, allows to take ready OB.

Some primitive throttling and garbage collection is required, to ensure that OBs are not executed many times and that pool doesn't OOM when there are forks. For that purpose, we maintain minimal_base_height for chunks and block_height_threshold for blocks. Note that we don't remove chunks immediately because if there is a block skip, chunks should be reused to process the next OB.

This feature is independent, so I also implement simple unit tests for it.

Processing OB

As discussed before, result of chunk execution on top of OB doesn't impact any part of block processing and doesn't persist anything. It is simply put to cache which can be reused when the actual block is received.

This cache, however, needs some unique key to store results. For that, I introduce CachedShardUpdateKey which includes necessary fields for Block or OB, all the chunks and shard id (index could also work). Note that we need chunk hashes because they define prev outgoing receipts, which in turn are used to generate incoming receipts for our chunk.

For execution, BlocksInProcessing is extended a bit to keep OBs as well to limit the number of parallel chunk executions for blocks and OBs together. The population happens in postprocess_optimistic_block.

Testing

Finally, we are also able to write test_optimistic_block. For now I just check that there is at least one cache hit, let's think about more complex cases later.

I'll resolve merge conflicts later.

@Longarithm Longarithm force-pushed the ob-pool-2 branch 3 times, most recently from 89a8294 to e8227d8 Compare January 24, 2025 14:32
@Longarithm Longarithm changed the title wip: optimistic block pool feat: optimistic block pool Feb 3, 2025
@Longarithm Longarithm marked this pull request as ready for review February 3, 2025 19:57
@Longarithm Longarithm requested a review from a team as a code owner February 3, 2025 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant