diff --git a/README.md b/README.md index 0ed076c..8ed0ac6 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,9 @@ Checkout the `Klife.Client` [docs](https://hexdocs.pm/klife/Klife.Client.html) f ## Producer performance -In order to test the performance I've prepared a benchmark that uses [benchee](https://github.com/bencheeorg/benchee) to produce +### Produce sync + +In order to test sync produce the performance I've prepared a benchmark that uses [benchee](https://github.com/bencheeorg/benchee) to produce kafka records on kafka cluster running locally on my machine. The details can be checked out on `benchmark.ex` mix task and the results on `bechmark_results`. @@ -117,7 +119,24 @@ The relevant client configuration are equal on all clients and they are: With this scenario I've executed the benchmark increasing the `parallel` attribute from benchee from 1 to 16, doubling it each round. The results are the following: -![](./assets/producer_sync_benchmark.png "Producer Benchmark Results") +![](./assets/producer_sync_benchmark.png "Producer Sync Benchmark Results") + +### Produce async + +The asynchronous benchmark spawns N parallel processes producing to one of 3 topics in a loop. After 10 seconds, +it calculates the difference between the initial and current offsets for each topic partition to determine +the total records produced and the throughput (records per second). + +The details can be checked out on `async_producer_benchmark.ex`. + +To reproduce it on your setup you can run (16 is the N value): + +``` +bash start-kafka.sh +mix benchmark producer_async 16 +``` + +![](./assets/producer_async_benchmark.png "Producer Async Benchmark Results") ## Compatibility with Kafka versions diff --git a/assets/producer_async_benchmark.png b/assets/producer_async_benchmark.png new file mode 100644 index 0000000..b675084 Binary files /dev/null and b/assets/producer_async_benchmark.png differ diff --git a/bechmark_results/32_async.txt b/bechmark_results/32_async.txt new file mode 100644 index 0000000..8388663 --- /dev/null +++ b/bechmark_results/32_async.txt @@ -0,0 +1,4 @@ +Client | Result | Compared to klife +brod | 619825 | x0.87 +erlkaf | 523386 | x0.74 +klife | 709398 | x1.0 diff --git a/config/config.exs b/config/config.exs index d3ac6e1..401db47 100644 --- a/config/config.exs +++ b/config/config.exs @@ -140,6 +140,40 @@ config :klife, MyClient, [name: "my_topic_3"] ] +# consumers: [ +# [ +# group_name: "klife_consumer_group_1", +# group_type: :consumer, +# topics: [ +# {"my_topic", MyHandler} +# ] +# ], +# [ +# group_name: "klife_share_group_1", +# group_type: :share, +# topics: [ +# {"my_topic", MyHandler}, +# {CustomTopicMatch, MyOtherHandler} +# ] +# ], +# [ +# group_name: "klife_classic_group_1", +# group_type: :classic, +# topics: [ +# {"my_topic", MyHandler}, +# {CustomTopicMatch, MyOtherHandler} +# ] +# ], +# [ +# group_name: "klife_classic_group_1", +# group_type: :classic, +# topics: [ +# {"my_topic", MyHandler}, +# {CustomTopicMatch, MyOtherHandler} +# ] +# ] +# ] + if config_env() == :dev do import_config "#{config_env()}.exs" end diff --git a/lib/klife/consumer/fetcher.ex b/lib/klife/consumer/fetcher.ex new file mode 100644 index 0000000..19ca31e --- /dev/null +++ b/lib/klife/consumer/fetcher.ex @@ -0,0 +1,3 @@ +# defmodule Klife.Consumer.Fetcher do +# use GenServer +# end