What is the optimal scheduler setting for RabbitMQ running in a cgroup

[EricLin89]

I'm having my rabbitmq-server process running on a linux environment with 2 cpu processor. Processor 1 has core 0 ~ 47, Processor 2 has core 48 ~ 95. The 2 processors are located in different numa nodes.
The rabbitmq-server process is constrained inside a cgroup, with cpu 2 ~ 41 assigned to it.
Running:

erlang:system_info(scheduler_bind_type)
erlang:system_info(scheduler_bindings)
erlang:system_info(schedulers)
erlang:system_info(schedulers_online)

with rabbitmqct eval gives me the following result:

thread_no_node_processor_spread
{2,3,4,...*skipped*...,40,41,unbound,unbound,...*skipped*...,unbound}
96
40

Version of rabbitmq-server is 3.7.13, the -stbt flag is set as db(default value).
Could anyone explain why the system_info query result is as above? What is the recommended configuration for such usage scenario(clearly the default setting is not optimal)?

[lukebakken]

Please provide your Erlang version.

[EricLin89]

erlang version is 21.3.8.6

[lukebakken]

While this is more of an "Erlang VM" question, here are some thoughts -

• You should try the absolute latest version of Erlang, 24.1.7. I'm pretty sure there have been changes since 21 that affect scheduler binding in constrained environments.
• Did you run RabbitMQ / Erlang outside of a cgroup to compare values?
• Do you see any issues with the current behavior?

[EricLin89]

@lukebakken Thanks.
After comparing with another RabbitMQ process not inside a cgroup, I figured out what the above output means:
erlang:system_info(scheduler_bindings) = {2,3,4,...*skipped*...,40,41,unbound,unbound,...*skipped*...,unbound}
means 40 of the schedulers are bound to core 2 ~41, others are unbound.

erlang:system_info(schedulers) = 96
erlang:system_info(schedulers_online) = 40

means there are totally 96 schedulers but only 40 of them are working.
Those seems reasonable.

I'm not seeing any particular issue with the current behavior, but I'm having my RabbitMQ running in a large deployment, with 30000+ connections and 40000+ queues. When publishers send a lot of fanout message to over 1000 consumers, I see a lot of "missed heartbeat" and connection lost errors, which I believe is very likely caused by RabbitMQ unable to handle such heavy load. I want to see if I can do some performance tuning to improve the situation.

[mkuratczyk]

It's likely possible to tune your configuration but it's not a trivial task and usually involves a lot of trial and error. Something you can consider is the Streams feature added in 3.9, which is great for large fan-out situations as messages don't have to be copied to each consumer's queue - they can be stored once in the stream and consumed multiple times by different clients: https://rabbitmq.com/streams.html