remove view_state limit
#145
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
frol
reviewed
Dec 13, 2023
Comment on lines
+51
to
+60
| for state_keys_chunk in state_keys.chunks(1000) { | ||
| // TODO: 1000 is hardcoded value. Make it configurable. | ||
| let mut tasks_futures = vec![]; | ||
| for state_key in state_keys_chunk { | ||
| let state_value_result_future = | ||
| db_manager.get_state_key_value(account_id, block_height, state_key.clone()); | ||
| tasks_futures.push(state_value_result_future); | ||
| } | ||
| let results = futures::future::join_all(tasks_futures).await; | ||
| for (state_key, state_value) in results.into_iter() { |
There was a problem hiding this comment.
FYI, this code will suffer from waiting the longest request out of 1000 in order to move on to the next batch of 1000. You can do better if you turn it into a stream of concurrent requests (FuturesUnordered can be the solution, but you don't really need to use it explicitly):
https://github.com/near/near-cli-rs/blob/7aca0acfe772fa7bc2c06c470e68ae8d05edd94f/src/commands/account/view_account_summary/mod.rs#L62-L82 (you don't need block_on here, obviously, use .await instead
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes Implemented
1. Removed Key Limit: The previously imposed limitation on the number of keys in a state has been eliminated. This means our system can now handle a larger volume of keys without restrictions.
2. Optimization of Queries: We've implemented optimizations for both key and value queries. This enhancement ensures improved efficiency and performance when retrieving information related to keys and their corresponding values.
Temporary Solution
It's important to note that while these changes represent a substantial improvement, this solution is temporary. Although the system now allows for handling a greater number of keys, there's no absolute guarantee for handling exceptionally large datasets, such as those in the gigabyte range.
Future Steps
1. Pagination Implementation: The upcoming step will introduce new endpoints designed explicitly for pagination. These endpoints will facilitate the handling of large states by allowing the data to be retrieved in manageable chunks or pages.
2. Guaranteed Retrieval for Large States: These new endpoints will guarantee the retrieval of all keys, even for states with massive data volumes, ensuring that clients can reliably access and process extensive datasets without constraints.
This pull request lays the groundwork for future improvements, acknowledging the current enhancements while also outlining a clear path for addressing the handling of exceptionally large datasets in the upcoming iterations of the system.