Skip to content

Commit

Permalink
chore: add async produce benchmark result
Browse files Browse the repository at this point in the history
  • Loading branch information
oliveigah committed Nov 18, 2024
1 parent 65bd8db commit ef45b74
Show file tree
Hide file tree
Showing 5 changed files with 62 additions and 2 deletions.
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ Checkout the `Klife.Client` [docs](https://hexdocs.pm/klife/Klife.Client.html) f

## Producer performance

In order to test the performance I've prepared a benchmark that uses [benchee](https://github.com/bencheeorg/benchee) to produce
### Produce sync

In order to test sync produce the performance I've prepared a benchmark that uses [benchee](https://github.com/bencheeorg/benchee) to produce
kafka records on kafka cluster running locally on my machine.

The details can be checked out on `benchmark.ex` mix task and the results on `bechmark_results`.
Expand Down Expand Up @@ -117,7 +119,24 @@ The relevant client configuration are equal on all clients and they are:
With this scenario I've executed the benchmark increasing the `parallel` attribute from
benchee from 1 to 16, doubling it each round. The results are the following:

![](./assets/producer_sync_benchmark.png "Producer Benchmark Results")
![](./assets/producer_sync_benchmark.png "Producer Sync Benchmark Results")

### Produce async

The asynchronous benchmark spawns N parallel processes producing to one of 3 topics in a loop. After 10 seconds,
it calculates the difference between the initial and current offsets for each topic partition to determine
the total records produced and the throughput (records per second).

The details can be checked out on `async_producer_benchmark.ex`.

To reproduce it on your setup you can run (16 is the N value):

```
bash start-kafka.sh
mix benchmark producer_async 16
```

![](./assets/producer_async_benchmark.png "Producer Async Benchmark Results")

## Compatibility with Kafka versions

Expand Down
Binary file added assets/producer_async_benchmark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions bechmark_results/32_async.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Client | Result | Compared to klife
brod | 619825 | x0.87
erlkaf | 523386 | x0.74
klife | 709398 | x1.0
34 changes: 34 additions & 0 deletions config/config.exs
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,40 @@ config :klife, MyClient,
[name: "my_topic_3"]
]

# consumers: [
# [
# group_name: "klife_consumer_group_1",
# group_type: :consumer,
# topics: [
# {"my_topic", MyHandler}
# ]
# ],
# [
# group_name: "klife_share_group_1",
# group_type: :share,
# topics: [
# {"my_topic", MyHandler},
# {CustomTopicMatch, MyOtherHandler}
# ]
# ],
# [
# group_name: "klife_classic_group_1",
# group_type: :classic,
# topics: [
# {"my_topic", MyHandler},
# {CustomTopicMatch, MyOtherHandler}
# ]
# ],
# [
# group_name: "klife_classic_group_1",
# group_type: :classic,
# topics: [
# {"my_topic", MyHandler},
# {CustomTopicMatch, MyOtherHandler}
# ]
# ]
# ]

if config_env() == :dev do
import_config "#{config_env()}.exs"
end
3 changes: 3 additions & 0 deletions lib/klife/consumer/fetcher.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# defmodule Klife.Consumer.Fetcher do
# use GenServer
# end

0 comments on commit ef45b74

Please sign in to comment.