Use futures in the Source poll call and logging enhancements #28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drnushooz
changed the title
|
Talked through a separate side channel -- I'm not convinced that this will solve the performance issues since the bottleneck doesn't appear to be related to transforming data and putting it in a queue. Basically, we need a way to better parallelize the connector so that keyspace notifications parallelize based on the node. The solution for that is dependent on a fix for #27. I'm also not convinced at a glance that this PR isn't dropping records once max poll is reached. More testing in this area is needed and I'm currently pre-occupied with other projects and personal stuff during my evening hours. I'm actively interested in improving the performance however and I'll file a new ticket for this. |
|
Filed #29 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The poll call in the source task today uses a single thread. From our internal tests, it has showed better performance when
CompletableFuturesare used at it distributes the load among multiple threads. In addition, there is also additional debug level logging to show which Redis node the data is coming from to debug any issues pertinent to data distribution in Redis. This logging has helped us internally to diagnose some problems. The PR also makesmaxPollSizea configurable parameter with the same default.