Change Kafka Buffer defaults for fetch.max.wait.ms, fetch.min.bytes, partition.assignment.strategy, close consumer on shutdown

[graytaylor0]

Description

When data prepper scales horizontally and removes data prepper instances, this can cause instability due to using the RangeAssignor, since the behavior for RangeAssignor is

When a rebalance occurs, all consumers drop all their partitions and get reassigned (a full rebalance).

CoopeartiveStickyAssignor will gradually release partitions instead of dropping them all at once and reassigning when a consumer leave the group

Additionally, the fetch.max.wait.ms at 500 ms and fetch.min.bytes as 1 will lead to many more calls to consume and return smaller number of records. Changing this to 1000 ms and 2 kb will lead to less calls to consume and more records being returned in each poll call, which will also lower the utilization on the kafka cluster.

Lastly, the Kafka buffer consumers were not calling consumer.close() on shutdown, which will mean that the Kafka server will wait for heartbeat to fail (around 45 seconds) before it realizes it has lost a consumer, rather than immediately.

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.