[Feature](parallel-result-sink) support async fetch from multiple backends concurrently #47915
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
1 similar comment
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
| ReceiverContext context = new ReceiverContext(resultReceivers.get(i), i); | ||
| contexts.add(context); | ||
| } | ||
| this.executor = Executors.newFixedThreadPool(resultReceivers.size()); |
There was a problem hiding this comment.
how about use a global thread pool with unlimited size?
to save thread creation time
There was a problem hiding this comment.
and you are not closing the thread pool?
| BlockingQueue<Integer> readyOffsets; | ||
| int finishedReceivers = 0; | ||
|
|
||
| public ResultReceiverConsumer(List<ResultReceiver> resultReceivers) { |
There was a problem hiding this comment.
I suggest to write a ut for this class
Here is a example written by cursor, I didn't not run it, but may help you
ResultReceiverConsumerTest.java.txt
What problem does this PR solve?
support async fetch from multiple backends concurrently
If we fetch data one by one with the backend in order, the logical deadlock may be triggered due to memory control, and the load on the backend is also unbalanced.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)