High cpu utilization in using flume kafka sink

[luyue0821]

Hi Hari,

I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu utilization stay at 100% and never dropped down all the time even at the time the channel is empty.

I looked into the source code and found that "process" function in KafkaSink always return Status.READY even if no events available in channel. That causes the SinkRunner keep running achieving event from channel and get nothing.

Do we need to change to return Status.BACKOFF in "process" function in KafkaSink when it notices that there is no events processed in current round? So that the SinkRunner has a chance to take a rest when there is no event in channel. If this proposal feasible, function "testEmptyChannel" in TestKafkaSink also need to be changed.

Thanks
Yue

[harishreedharan]

hi @luyue0821, can you please file a jira here: https://issues.apache.org/jira/browse/FLUME

We don't use github for tracking issues or development. This is my fork of the actual repo that lives here: https://git-wip-us.apache.org/repos/asf?p=flume.git

Since you also know the actual cause, which does seem like a valid bug, do you want to submit a patch?

[Mercedechf]

hi @luyue0821 ,We found that one of our flume's cpu utilization also stay at 100%, We also use kafka sink of flume 1.6.But I can't reproduce the issue. And the kafkasink only take event from channel about every 3 seconds. Can you paste your flume config file or tell how to reproduce the issue? thanks very much.