Issue while creating additional users using Helm chart on Azure Kubernetes Service

[1993Saurabh]

Hi,
I am currently having an issue while creating the users using Helm charts and Azure Kubernetes Services who can login into the RabbitMQ management console.

I am using the load_definitions.json file to load it into the Kubernetes secret and then providing the file path to overrides file under extraConfiguration. Below is the content of both the files I am using currently to install the RabbitMQ using Helm chart:

overrides.yaml

image:
registry: azure-container-registry.azurecr.io
repository: bitnami/rabbitmq
tag: 3.8.2-debian-10-r30

metrics:
image:
registry: azure-container-registry.azurecr.io
repository: bitnami/rabbitmq-exporter
tag: 0.29.0-debian-10-r28

volumePermissions:
image:
registry: azure-container-registry.azurecr.io
repository: bitnami/minideb
tag: buster

loadDefinition:
enabled: true
existingSecret: "rabbitmq-load-definition"

extraConfiguration: |
load_definitions = /app/load_definition.json

auth:
password: password
erlangCookie: OYs54TlVmtxqdaDcu7xc12syQTrZ0nt6

load_definition.json

{
"users": [
{
"name": "saurabh",
"password": "Passw0rd12",
"tags": "administrator"
},
{
"name": "saurabh1",
"password": "Passw0rd12",
"tags": "management"
}
],
"vhosts": [
{
"name": "/"
},
{
"name": "/myvhost1"
}
],
"permissions":[
{
"user": "saurabh",
"vhost": "/",
"configure": ".",
"write": ".",
"read": "."
},
{
"user": "saurabh1",
"vhost": "/myvhost1",
"configure": ".",
"write": ".",
"read": "."
}
]
}

The users are not getting created when I install the helm chart in Azure Kubernetes Service cluster. Even though I am not able to login into the RabbitMQ management console using the default user's credentials.

However when I create the users using the following three commands, I am successfully able to login into the console using the below credentials:

rabbitmqctl add_user test test
rabbitmqctl set_user_tags test administrator
rabbitmqctl set_permissions -p / test "." "." ".*"

I am not sure where I am doing wrong. May be if anyone out there can help me out in this regard, I would be really grateful.
Any reference or the sample files would be really helpful too.

[michaelklishin]

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. This assumes that we have a certain amount of information to work with.

Getting all the details necessary to reproduce an issue, make a conclusion or even form a hypothesis about what's happening can take a fair amount of time. Our team is multiple orders of magnitude smaller than the RabbitMQ community. Please help others help you by providing a way to reproduce the behavior you're
observing and sharing as much relevant information as possible on the list:

• Server, client library and plugin (if applicable) versions used
• Server logs
• A code example or terminal transcript that can be used to reproduce
• Full exception stack traces (not a single line message)
• rabbitmqctl status (and, if possible, rabbitmqctl environment output)
• Other relevant things about the environment and workload, e.g. OS/systemd logs, a traffic capture, deployment tool logs, etc

Feel free to edit out hostnames and other potentially sensitive information.

When/if we have a complete enough understanding of what's going on, a recommendation will be provided or a new issues with more context will be filed.

Thank you.

[michaelklishin]

You are using a version of RabbitMQ that is 7 patch releases behind and a legacy way of deploying it Kubernetes (the Operator is the modern way).

While nothing has changed specifically w.r.t. how the users are imported since 3.8.2, management.load_definitions has been superseded by load_definitions which does not have some pretty important limitations.

We cannot suggest much without having an actual rabbitmq.conf file and full logs from node startup. We do not guess in this community: guessing is a very expensive approach to troublshooting and you are asking for free support here.

[1993Saurabh]

Hi @michaelklishin,

Thank you for your response.

I would like to know if I am deploying the RabbitMQ in a legacy way using the helm chart and if it is a 7 patch releases behind then why Bitnami haven't upgraded their repository which is actually now maintaining the RabbitMQ helm chart.

Logs fetched from Kubernetes:

2020-12-21T11:19:42.511194620Z �[38;5;6m �[38;5;5m11:19:42.50 �[0m
2020-12-21T11:19:42.511662220Z �[38;5;6m �[38;5;5m11:19:42.51 �[0m�[1mWelcome to the Bitnami rabbitmq container�[0m
2020-12-21T11:19:42.512884018Z �[38;5;6m �[38;5;5m11:19:42.51 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/bitnami-docker-rabbitmq�[0m
2020-12-21T11:19:42.514257016Z �[38;5;6m �[38;5;5m11:19:42.51 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/bitnami-docker-rabbitmq/issues�[0m
2020-12-21T11:19:42.515416714Z �[38;5;6m �[38;5;5m11:19:42.51 �[0mSend us your feedback at �[[email protected]�[0m
2020-12-21T11:19:42.516719912Z �[38;5;6m �[38;5;5m11:19:42.51 �[0m
2020-12-21T11:19:42.518134909Z �[38;5;6m �[38;5;5m11:19:42.51 �[0m�[38;5;2mINFO �[0m ==> ** Starting RabbitMQ setup **
2020-12-21T11:19:42.527261495Z �[38;5;6m �[38;5;5m11:19:42.52 �[0m�[38;5;2mINFO �[0m ==> Validating settings in RABBITMQ_* env vars..
2020-12-21T11:19:42.535593182Z �[38;5;6m �[38;5;5m11:19:42.53 �[0m�[38;5;2mINFO �[0m ==> Initializing RabbitMQ...
2020-12-21T11:19:42.540851874Z �[38;5;6m �[38;5;5m11:19:42.54 �[0m�[38;5;3mWARN �[0m ==> Configuration files won't be persisted anymore!
2020-12-21T11:19:42.544218368Z cp: cannot stat '/bitnami/rabbitmq/var/lib/rabbitmq/mnesia': No such file or directory
2020-12-21T11:19:42.547810363Z �[38;5;6m �[38;5;5m11:19:42.54 �[0m�[38;5;3mWARN �[0m ==> Old configuration migrated, please manually remove the 'conf', 'var' and '.rabbitmq' directories from the volume.
2020-12-21T11:19:42.566322833Z �[38;5;6m �[38;5;5m11:19:42.56 �[0m�[38;5;2mINFO �[0m ==> Persisted data detected. Restoring...
2020-12-21T11:19:44.448602961Z �[38;5;6m �[38;5;5m11:19:44.44 �[0m�[38;5;2mINFO �[0m ==> ** RabbitMQ setup finished! **
2020-12-21T11:19:44.448712861Z
2020-12-21T11:19:44.458442145Z �[38;5;6m �[38;5;5m11:19:44.45 �[0m�[38;5;2mINFO �[0m ==> ** Starting RabbitMQ **
2020-12-21T11:19:44.505477971Z 2020-12-21 11:19:44 [warning] Both old (.config) and new (.conf) format config files exist.
2020-12-21T11:19:44.505527271Z Using the old format config file: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config
2020-12-21T11:19:44.505533271Z Please update your config files to the new format and remove the old file.
2020-12-21T11:19:51.170163848Z 2020-12-21 11:19:51.169 [info] <0.8.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.170321847Z 2020-12-21 11:19:51.169 [info] <0.8.0> Feature flags: [ ] drop_unroutable_metric
2020-12-21T11:19:51.170442247Z 2020-12-21 11:19:51.170 [info] <0.8.0> Feature flags: [ ] empty_basic_get_metric
2020-12-21T11:19:51.170453347Z 2020-12-21 11:19:51.170 [info] <0.8.0> Feature flags: [ ] implicit_default_bindings
2020-12-21T11:19:51.170554947Z 2020-12-21 11:19:51.170 [info] <0.8.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.170766747Z 2020-12-21 11:19:51.170 [info] <0.8.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.170776447Z 2020-12-21 11:19:51.170 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.210748784Z 2020-12-21 11:19:51.210 [info] <0.268.0> ra: meta data store initialised. 0 record(s) recovered
2020-12-21T11:19:51.211213583Z 2020-12-21 11:19:51.211 [info] <0.273.0> WAL: recovering []
2020-12-21T11:19:51.212137981Z 2020-12-21 11:19:51.211 [info] <0.277.0>
2020-12-21T11:19:51.212148481Z Starting RabbitMQ 3.8.2 on Erlang 22.2
2020-12-21T11:19:51.212153181Z Copyright (c) 2007-2019 Pivotal Software, Inc.
2020-12-21T11:19:51.212157281Z Licensed under the MPL 1.1. Website: https://rabbitmq.com
2020-12-21T11:19:51.212561881Z
2020-12-21T11:19:51.212573981Z ## ## RabbitMQ 3.8.2
2020-12-21T11:19:51.212577981Z ## ##
2020-12-21T11:19:51.212591981Z ########## Copyright (c) 2007-2019 Pivotal Software, Inc.
2020-12-21T11:19:51.212596381Z ###### ##
2020-12-21T11:19:51.212622381Z ########## Licensed under the MPL 1.1. Website: https://rabbitmq.com
2020-12-21T11:19:51.212628181Z
2020-12-21T11:19:51.212650081Z Doc guides: https://rabbitmq.com/documentation.html
2020-12-21T11:19:51.212654181Z Support: https://rabbitmq.com/contact.html
2020-12-21T11:19:51.212658481Z Tutorials: https://rabbitmq.com/getstarted.html
2020-12-21T11:19:51.212662581Z Monitoring: https://rabbitmq.com/monitoring.html
2020-12-21T11:19:51.212666081Z
2020-12-21T11:19:51.212669781Z Logs:
2020-12-21T11:19:51.212673580Z
2020-12-21T11:19:51.212677380Z Config file(s): /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config
2020-12-21T11:19:51.212681580Z
2020-12-21T11:19:51.213435679Z Starting broker...2020-12-21 11:19:51.212 [info] <0.277.0>
2020-12-21T11:19:51.213447979Z node : [email protected]
2020-12-21T11:19:51.213454379Z home dir : /opt/bitnami/rabbitmq/.rabbitmq
2020-12-21T11:19:51.213459079Z config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config
2020-12-21T11:19:51.213463279Z cookie hash : xchxgBcwj0X7k+1LwJHnyA==
2020-12-21T11:19:51.213467379Z log(s) :
2020-12-21T11:19:51.213497179Z database dir : /bitnami/rabbitmq/mnesia/[email protected]
2020-12-21T11:19:51.226460359Z 2020-12-21 11:19:51.226 [info] <0.277.0> Running boot step pre_boot defined by app rabbit
2020-12-21T11:19:51.227018158Z 2020-12-21 11:19:51.226 [info] <0.277.0> Running boot step rabbit_core_metrics defined by app rabbit
2020-12-21T11:19:51.227206258Z 2020-12-21 11:19:51.226 [info] <0.277.0> Running boot step rabbit_alarm defined by app rabbit
2020-12-21T11:19:51.231640751Z 2020-12-21 11:19:51.231 [info] <0.283.0> Memory high watermark set to 1574 MiB (1651041894 bytes) of 3936 MiB (4127604736 bytes) total
2020-12-21T11:19:51.236143943Z 2020-12-21 11:19:51.236 [info] <0.285.0> Enabling free disk space monitoring
2020-12-21T11:19:51.236470043Z 2020-12-21 11:19:51.236 [info] <0.285.0> Disk free limit set to 4127MB
2020-12-21T11:19:51.239554238Z 2020-12-21 11:19:51.239 [info] <0.277.0> Running boot step code_server_cache defined by app rabbit
2020-12-21T11:19:51.239682138Z 2020-12-21 11:19:51.239 [info] <0.277.0> Running boot step file_handle_cache defined by app rabbit
2020-12-21T11:19:51.239964937Z 2020-12-21 11:19:51.239 [info] <0.288.0> Limiting to approx 1048479 file handles (943629 sockets)
2020-12-21T11:19:51.240104237Z 2020-12-21 11:19:51.239 [info] <0.289.0> FHC read buffering: OFF
2020-12-21T11:19:51.240113337Z 2020-12-21 11:19:51.239 [info] <0.289.0> FHC write buffering: ON
2020-12-21T11:19:51.240378937Z 2020-12-21 11:19:51.240 [info] <0.277.0> Running boot step worker_pool defined by app rabbit
2020-12-21T11:19:51.240521637Z 2020-12-21 11:19:51.240 [info] <0.278.0> Will use 2 processes for default worker pool
2020-12-21T11:19:51.240530837Z 2020-12-21 11:19:51.240 [info] <0.278.0> Starting worker pool 'worker_pool' with 2 processes in it
2020-12-21T11:19:51.240762336Z 2020-12-21 11:19:51.240 [info] <0.277.0> Running boot step database defined by app rabbit
2020-12-21T11:19:51.241108036Z 2020-12-21 11:19:51.240 [info] <0.277.0> Node database directory at /bitnami/rabbitmq/mnesia/[email protected] is empty. Assuming we need to join an existing cluster or initialise from scratch...
2020-12-21T11:19:51.241250535Z 2020-12-21 11:19:51.241 [info] <0.277.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2020-12-21T11:19:51.241259235Z 2020-12-21 11:19:51.241 [info] <0.277.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2020-12-21T11:19:51.241272635Z 2020-12-21 11:19:51.241 [info] <0.277.0> Peer discovery backend does not support locking, falling back to randomized delay
2020-12-21T11:19:51.241417535Z 2020-12-21 11:19:51.241 [info] <0.277.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2020-12-21T11:19:51.241426235Z 2020-12-21 11:19:51.241 [info] <0.277.0> All discovered existing cluster peers:
2020-12-21T11:19:51.241430735Z 2020-12-21 11:19:51.241 [info] <0.277.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
2020-12-21T11:19:51.243176632Z 2020-12-21 11:19:51.242 [info] <0.43.0> Application mnesia exited with reason: stopped
2020-12-21T11:19:51.429871338Z 2020-12-21 11:19:51.429 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.465612881Z 2020-12-21 11:19:51.465 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.466070380Z 2020-12-21 11:19:51.465 [info] <0.277.0> Feature flag drop_unroutable_metric: supported, attempt to enable...
2020-12-21T11:19:51.466323180Z 2020-12-21 11:19:51.465 [info] <0.277.0> Feature flag drop_unroutable_metric: mark as enabled=state_changing
2020-12-21T11:19:51.476697964Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.476874563Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: [] drop_unroutable_metric
2020-12-21T11:19:51.476883663Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: [ ] empty_basic_get_metric
2020-12-21T11:19:51.476887763Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: [ ] implicit_default_bindings
2020-12-21T11:19:51.476892063Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.477112763Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.477120863Z 2020-12-21 11:19:51.476 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.486399348Z 2020-12-21 11:19:51.486 [info] <0.277.0> Feature flag drop_unroutable_metric: mark as enabled=true
2020-12-21T11:19:51.499841927Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.500005627Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.500014727Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: [ ] empty_basic_get_metric
2020-12-21T11:19:51.500018627Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: [ ] implicit_default_bindings
2020-12-21T11:19:51.500024227Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.500155527Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.500164127Z 2020-12-21 11:19:51.499 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.511482509Z 2020-12-21 11:19:51.511 [info] <0.277.0> Feature flag empty_basic_get_metric: supported, attempt to enable...
2020-12-21T11:19:51.511686008Z 2020-12-21 11:19:51.511 [info] <0.277.0> Feature flag empty_basic_get_metric: mark as enabled=state_changing
2020-12-21T11:19:51.522901591Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.523078890Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.523233790Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: [] empty_basic_get_metric
2020-12-21T11:19:51.523399690Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: [ ] implicit_default_bindings
2020-12-21T11:19:51.523557890Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.523712989Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.523824189Z 2020-12-21 11:19:51.522 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.531705577Z 2020-12-21 11:19:51.531 [info] <0.277.0> Feature flag empty_basic_get_metric: mark as enabled=true
2020-12-21T11:19:51.543345958Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.543457758Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.543466158Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.543470158Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: [ ] implicit_default_bindings
2020-12-21T11:19:51.543572858Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.543598458Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.543602358Z 2020-12-21 11:19:51.543 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.554269541Z 2020-12-21 11:19:51.553 [info] <0.277.0> Feature flag implicit_default_bindings: supported, attempt to enable...
2020-12-21T11:19:51.554441141Z 2020-12-21 11:19:51.554 [info] <0.277.0> Feature flag implicit_default_bindings: mark as enabled=state_changing
2020-12-21T11:19:51.564827525Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.565013624Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.565169324Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.565298624Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: [] implicit_default_bindings
2020-12-21T11:19:51.565416224Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.565692523Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.565862123Z 2020-12-21 11:19:51.564 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.574605109Z 2020-12-21 11:19:51.574 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2020-12-21T11:19:51.576122607Z 2020-12-21 11:19:51.574 [info] <0.277.0> Feature flag implicit_default_bindings: mark as enabled=true
2020-12-21T11:19:51.588156688Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.588170088Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.588174688Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.588178488Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: [x] implicit_default_bindings
2020-12-21T11:19:51.588182888Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: [ ] quorum_queue
2020-12-21T11:19:51.588186988Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.588190588Z 2020-12-21 11:19:51.587 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.598106372Z 2020-12-21 11:19:51.597 [info] <0.277.0> Feature flag quorum_queue: supported, attempt to enable...
2020-12-21T11:19:51.598302972Z 2020-12-21 11:19:51.597 [info] <0.277.0> Feature flag quorum_queue: mark as enabled=state_changing
2020-12-21T11:19:51.609553954Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.609572654Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.609742554Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.609858053Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: [x] implicit_default_bindings
2020-12-21T11:19:51.609869753Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: [] quorum_queue
2020-12-21T11:19:51.610045653Z 2020-12-21 11:19:51.609 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.610207353Z 2020-12-21 11:19:51.610 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.619597738Z 2020-12-21 11:19:51.619 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.619814838Z 2020-12-21 11:19:51.619 [info] <0.277.0> Feature flag quorum_queue: migrating Mnesia table rabbit_queue...
2020-12-21T11:19:51.641390204Z 2020-12-21 11:19:51.641 [info] <0.277.0> Feature flag quorum_queue: migrating Mnesia table rabbit_durable_queue...
2020-12-21T11:19:51.666326764Z 2020-12-21 11:19:51.666 [info] <0.277.0> Feature flag quorum_queue: Mnesia tables migration done
2020-12-21T11:19:51.666510464Z 2020-12-21 11:19:51.666 [info] <0.277.0> Feature flag quorum_queue: mark as enabled=true
2020-12-21T11:19:51.678336145Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.678430345Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.678561945Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.678785245Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: [x] implicit_default_bindings
2020-12-21T11:19:51.678855144Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: [x] quorum_queue
2020-12-21T11:19:51.678987144Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: [ ] virtual_host_metadata
2020-12-21T11:19:51.679129444Z 2020-12-21 11:19:51.678 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.688107130Z 2020-12-21 11:19:51.687 [info] <0.277.0> Feature flag virtual_host_metadata: supported, attempt to enable...
2020-12-21T11:19:51.688366929Z 2020-12-21 11:19:51.688 [info] <0.277.0> Feature flag virtual_host_metadata: mark as enabled=state_changing
2020-12-21T11:19:51.699080513Z 2020-12-21 11:19:51.698 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.699129912Z 2020-12-21 11:19:51.698 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.699337812Z 2020-12-21 11:19:51.699 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.699380112Z 2020-12-21 11:19:51.699 [info] <0.277.0> Feature flags: [x] implicit_default_bindings
2020-12-21T11:19:51.699534612Z 2020-12-21 11:19:51.699 [info] <0.277.0> Feature flags: [x] quorum_queue
2020-12-21T11:19:51.699758211Z 2020-12-21 11:19:51.699 [info] <0.277.0> Feature flags: [~] virtual_host_metadata
2020-12-21T11:19:51.699903511Z 2020-12-21 11:19:51.699 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.708307598Z 2020-12-21 11:19:51.708 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.737316752Z 2020-12-21 11:19:51.737 [info] <0.277.0> Feature flag virtual_host_metadata: mark as enabled=true
2020-12-21T11:19:51.750312032Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: list of feature flags found:
2020-12-21T11:19:51.750443031Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: [x] drop_unroutable_metric
2020-12-21T11:19:51.750553031Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: [x] empty_basic_get_metric
2020-12-21T11:19:51.750641231Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: [x] implicit_default_bindings
2020-12-21T11:19:51.750725331Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: [x] quorum_queue
2020-12-21T11:19:51.750809631Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: [x] virtual_host_metadata
2020-12-21T11:19:51.750890231Z 2020-12-21 11:19:51.750 [info] <0.277.0> Feature flags: feature flag states written to disk: yes
2020-12-21T11:19:51.761734714Z 2020-12-21 11:19:51.761 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.799191454Z 2020-12-21 11:19:51.799 [info] <0.277.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-12-21T11:19:51.799406254Z 2020-12-21 11:19:51.799 [info] <0.277.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2020-12-21T11:19:51.799580254Z 2020-12-21 11:19:51.799 [info] <0.277.0> Running boot step database_sync defined by app rabbit
2020-12-21T11:19:51.799762454Z 2020-12-21 11:19:51.799 [info] <0.277.0> Running boot step feature_flags defined by app rabbit
2020-12-21T11:19:51.800184453Z 2020-12-21 11:19:51.799 [info] <0.277.0> Running boot step codec_correctness_check defined by app rabbit
2020-12-21T11:19:51.800341953Z 2020-12-21 11:19:51.799 [info] <0.277.0> Running boot step external_infrastructure defined by app rabbit
2020-12-21T11:19:51.800350853Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_registry defined by app rabbit
2020-12-21T11:19:51.800355453Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2020-12-21T11:19:51.800477752Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_queue_location_random defined by app rabbit
2020-12-21T11:19:51.800673452Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_event defined by app rabbit
2020-12-21T11:19:51.800855452Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2020-12-21T11:19:51.801032952Z 2020-12-21 11:19:51.800 [info] <0.277.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2020-12-21T11:19:51.801315151Z 2020-12-21 11:19:51.801 [info] <0.277.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2020-12-21T11:19:51.801513751Z 2020-12-21 11:19:51.801 [info] <0.277.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2020-12-21T11:19:51.801738050Z 2020-12-21 11:19:51.801 [info] <0.277.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2020-12-21T11:19:51.802043850Z 2020-12-21 11:19:51.801 [info] <0.277.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2020-12-21T11:19:51.802237750Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2020-12-21T11:19:51.802433649Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2020-12-21T11:19:51.802615449Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2020-12-21T11:19:51.802784149Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_priority_queue defined by app rabbit
2020-12-21T11:19:51.802908149Z 2020-12-21 11:19:51.802 [info] <0.277.0> Priority queues enabled, real BQ is rabbit_variable_queue
2020-12-21T11:19:51.803028448Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2020-12-21T11:19:51.803200148Z 2020-12-21 11:19:51.802 [info] <0.277.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2020-12-21T11:19:51.803484948Z 2020-12-21 11:19:51.803 [info] <0.277.0> Running boot step kernel_ready defined by app rabbit
2020-12-21T11:19:51.803974247Z 2020-12-21 11:19:51.803 [info] <0.277.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2020-12-21T11:19:51.804118347Z 2020-12-21 11:19:51.803 [info] <0.277.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2020-12-21T11:19:51.808867439Z 2020-12-21 11:19:51.808 [info] <0.517.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2020-12-21T11:19:51.808881739Z 2020-12-21 11:19:51.808 [info] <0.277.0> Running boot step guid_generator defined by app rabbit
2020-12-21T11:19:51.815448429Z 2020-12-21 11:19:51.815 [info] <0.277.0> Running boot step rabbit_node_monitor defined by app rabbit
2020-12-21T11:19:51.815780128Z 2020-12-21 11:19:51.815 [info] <0.523.0> Starting rabbit_node_monitor
2020-12-21T11:19:51.815943428Z 2020-12-21 11:19:51.815 [info] <0.277.0> Running boot step delegate_sup defined by app rabbit
2020-12-21T11:19:51.816694727Z 2020-12-21 11:19:51.816 [info] <0.277.0> Running boot step rabbit_memory_monitor defined by app rabbit
2020-12-21T11:19:51.816937626Z 2020-12-21 11:19:51.816 [info] <0.277.0> Running boot step core_initialized defined by app rabbit
2020-12-21T11:19:51.817133126Z 2020-12-21 11:19:51.816 [info] <0.277.0> Running boot step upgrade_queues defined by app rabbit
2020-12-21T11:19:51.839260691Z 2020-12-21 11:19:51.839 [info] <0.277.0> message_store upgrades: 1 to apply
2020-12-21T11:19:51.839573091Z 2020-12-21 11:19:51.839 [info] <0.277.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2020-12-21T11:19:51.839906390Z 2020-12-21 11:19:51.839 [info] <0.277.0> message_store upgrades: No durable queues found. Skipping message store migration
2020-12-21T11:19:51.840032490Z 2020-12-21 11:19:51.839 [info] <0.277.0> message_store upgrades: Removing the old message store data
2020-12-21T11:19:51.840407189Z 2020-12-21 11:19:51.840 [info] <0.277.0> message_store upgrades: All upgrades applied successfully
2020-12-21T11:19:51.867493247Z 2020-12-21 11:19:51.867 [info] <0.277.0> Running boot step rabbit_connection_tracking defined by app rabbit
2020-12-21T11:19:51.867707946Z 2020-12-21 11:19:51.867 [info] <0.277.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2020-12-21T11:19:51.867845346Z 2020-12-21 11:19:51.867 [info] <0.277.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2020-12-21T11:19:51.868052846Z 2020-12-21 11:19:51.867 [info] <0.277.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2020-12-21T11:19:51.868465545Z 2020-12-21 11:19:51.868 [info] <0.277.0> Running boot step rabbit_policies defined by app rabbit
2020-12-21T11:19:51.869414544Z 2020-12-21 11:19:51.869 [info] <0.277.0> Running boot step rabbit_policy defined by app rabbit
2020-12-21T11:19:51.869547943Z 2020-12-21 11:19:51.869 [info] <0.277.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2020-12-21T11:19:51.869820343Z 2020-12-21 11:19:51.869 [info] <0.277.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2020-12-21T11:19:51.870063743Z 2020-12-21 11:19:51.869 [info] <0.277.0> Running boot step rabbit_vhost_limit defined by app rabbit
2020-12-21T11:19:51.870348742Z 2020-12-21 11:19:51.870 [info] <0.277.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2020-12-21T11:19:51.870544942Z 2020-12-21 11:19:51.870 [info] <0.277.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2020-12-21T11:19:51.870733942Z 2020-12-21 11:19:51.870 [info] <0.277.0> Management plugin: using rates mode 'basic'
2020-12-21T11:19:51.871364841Z 2020-12-21 11:19:51.871 [info] <0.277.0> Running boot step recovery defined by app rabbit
2020-12-21T11:19:51.872209839Z 2020-12-21 11:19:51.872 [info] <0.277.0> Running boot step load_core_definitions defined by app rabbit
2020-12-21T11:19:51.872407439Z 2020-12-21 11:19:51.872 [info] <0.277.0> Running boot step empty_db_check defined by app rabbit
2020-12-21T11:19:51.872525539Z 2020-12-21 11:19:51.872 [info] <0.277.0> Adding vhost '/' (description: 'Default virtual host')
2020-12-21T11:19:51.915300671Z 2020-12-21 11:19:51.915 [info] <0.564.0> Making sure data directory '/bitnami/rabbitmq/mnesia/[email protected]/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2020-12-21T11:19:51.921603661Z 2020-12-21 11:19:51.921 [info] <0.564.0> Starting message stores for vhost '/'
2020-12-21T11:19:51.921956161Z 2020-12-21 11:19:51.921 [info] <0.568.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2020-12-21T11:19:51.923574458Z 2020-12-21 11:19:51.923 [info] <0.564.0> Started message store of type transient for vhost '/'
2020-12-21T11:19:51.923911358Z 2020-12-21 11:19:51.923 [info] <0.571.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2020-12-21T11:19:51.924865856Z 2020-12-21 11:19:51.924 [warning] <0.571.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2020-12-21T11:19:51.926721953Z 2020-12-21 11:19:51.926 [info] <0.564.0> Started message store of type persistent for vhost '/'
2020-12-21T11:19:51.928805150Z 2020-12-21 11:19:51.928 [info] <0.277.0> Creating user 'saurabh'
2020-12-21T11:19:51.935331640Z 2020-12-21 11:19:51.935 [info] <0.277.0> Setting user tags for user 'saurabh' to [administrator]
2020-12-21T11:19:51.940037832Z 2020-12-21 11:19:51.939 [info] <0.277.0> Setting permissions for 'saurabh' in '/' to '.', '.', '.*'
2020-12-21T11:19:51.945099024Z 2020-12-21 11:19:51.944 [info] <0.277.0> Running boot step rabbit_looking_glass defined by app rabbit
2020-12-21T11:19:51.945247624Z 2020-12-21 11:19:51.944 [info] <0.277.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2020-12-21T11:19:51.945769423Z 2020-12-21 11:19:51.945 [info] <0.277.0> Running boot step background_gc defined by app rabbit
2020-12-21T11:19:51.946353022Z 2020-12-21 11:19:51.946 [info] <0.277.0> Running boot step connection_tracking defined by app rabbit
2020-12-21T11:19:51.954918209Z 2020-12-21 11:19:51.954 [info] <0.277.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local'
2020-12-21T11:19:51.961180899Z 2020-12-21 11:19:51.961 [info] <0.277.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local'
2020-12-21T11:19:51.961558098Z 2020-12-21 11:19:51.961 [info] <0.277.0> Running boot step routing_ready defined by app rabbit
2020-12-21T11:19:51.961703298Z 2020-12-21 11:19:51.961 [info] <0.277.0> Running boot step pre_flight defined by app rabbit
2020-12-21T11:19:51.961906698Z 2020-12-21 11:19:51.961 [info] <0.277.0> Running boot step notify_cluster defined by app rabbit
2020-12-21T11:19:51.961931198Z 2020-12-21 11:19:51.961 [info] <0.277.0> Running boot step networking defined by app rabbit
2020-12-21T11:19:51.963943494Z 2020-12-21 11:19:51.963 [info] <0.617.0> started TCP listener on [::]:5672
2020-12-21T11:19:51.964258694Z 2020-12-21 11:19:51.964 [info] <0.277.0> Running boot step cluster_name defined by app rabbit
2020-12-21T11:19:51.964434594Z 2020-12-21 11:19:51.964 [info] <0.277.0> Running boot step direct_client defined by app rabbit
2020-12-21T11:19:51.991787650Z 2020-12-21 11:19:51.991 [info] <0.667.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-12-21T11:19:51.991987850Z 2020-12-21 11:19:51.991 [info] <0.773.0> Statistics database started.
2020-12-21T11:19:51.992139250Z 2020-12-21 11:19:51.991 [info] <0.772.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2020-12-21T11:19:52.134720625Z 2020-12-21 11:19:52.134 [info] <0.8.0> Server startup complete; 3 plugins started.
2020-12-21T11:19:52.134746425Z * rabbitmq_management
2020-12-21T11:19:52.134751625Z * rabbitmq_web_dispatch
2020-12-21T11:19:52.134757425Z * rabbitmq_management_agent
2020-12-21T11:19:52.135039324Z completed with 3 plugins.

[michaelklishin]

The Bitnami chart itself is years old. The Operator has gone 1.0 about a month ago. It's not clear to us what the future is for the Bitnami chart and our team does not maintain it. My best guess is that it won't be updated and will be superseded by a different Helm chart, if there's enough interest in having a Helm option.

I cannot comment on why the Bitnami chart still provides RabbitMQ 3.8.2. Again, our team does not maintain it and that project has a multi-year history of independent development and decisions.

[michaelklishin]

Looks like the chart tries to be smart about a configuration migration of some kind, from rabbitmq.config to rabbitmq.conf. Then the node logs the following:

2020-12-21T11:19:44.505477971Z 2020-12-21 11:19:44 [warning] Both old (.config) and new (.conf) format config files exist.
2020-12-21T11:19:44.505527271Z Using the old format config file: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config
2020-12-21T11:19:44.505533271Z Please update your config files to the new format and remove the old file.

So you may end up having two files without realizing it.

Here is what users are created on node start:

2020-12-21T11:19:51.928805150Z 2020-12-21 11:19:51.928 [info] <0.277.0> Creating user 'saurabh'
2020-12-21T11:19:51.935331640Z 2020-12-21 11:19:51.935 [info] <0.277.0> Setting user tags for user 'saurabh' to [administrator]
2020-12-21T11:19:51.940037832Z 2020-12-21 11:19:51.939 [info] <0.277.0> Setting permissions for 'saurabh' in '/' to '.', '.', '.*'

I don't see any evidence of definition import on this node. See the effective node configuration to see what config file was actually loaded and what settings it has. Again, no need to guess, the node can provide a lot of relevant information.

[1993Saurabh]

I can see it is using rabbitmq.conf file where path to load_definition.json is mentioned:

[michaelklishin]

There is a definition import example for the Operator. Enabling debug logging will provide a lot of relevant information about what exactly happens on boot and during sign in attempts.

[mkuratczyk]

While using the Operator is recommended, Bitnami does provide RabbitMQ 3.8.9: https://github.com/bitnami/charts/blob/master/bitnami/rabbitmq/Chart.yaml#L4. I'm not sure where you get your charts from but I guess you clone/mirror is not updated.

[1993Saurabh]

This actually works partially, though I had updated the repo but didn't realize that it is running the older version of RabbitMQ. I again added the repo and update it. Then I install it again but now facing this issue now:

I saw the log and can see there is an entry which states that the load_definiton is being imported but then got the above error.

Any help in this regard would be helpful.

[michaelklishin]

Here's what a 3.8.9 node will log when it imports definitions, including users:

2020-12-21 21:35:11.816 [info] <0.23994.0> Asked to import definitions. Acting user: rmq-internal
2020-12-21 21:35:11.816 [info] <0.23994.0> Importing concurrently 4 users...
2020-12-21 21:35:11.933 [info] <0.726.0> Successfully set user tags for user 'rabbithole_with_hashed_password1' to [policymaker,management]
2020-12-21 21:35:11.957 [info] <0.727.0> Successfully set user tags for user 'guest' to [administrator]
2020-12-21 21:35:11.957 [info] <0.728.0> Successfully set user tags for user 'alt-user1' to [administrator]
2020-12-21 21:35:11.957 [info] <0.729.0> Successfully set user tags for user 'tagged-user' to [monitoring,policymaker,impersonator]
2020-12-21 21:35:11.957 [info] <0.23994.0> Importing concurrently 4 vhosts...
2020-12-21 21:35:11.958 [info] <0.23994.0> Importing concurrently 5 permissions...
2020-12-21 21:35:11.982 [info] <0.727.0> Successfully set permissions for 'guest' in virtual host 'vh1' to '.*', '.*', '.*'
2020-12-21 21:35:11.982 [info] <0.726.0> Successfully set permissions for 'guest' in virtual host 'vh2' to '.*', '.*', '.*'
2020-12-21 21:35:12.001 [info] <0.729.0> Successfully set permissions for 'rabbithole_with_hashed_password1' in virtual host '/' to 'log.*', 'log.*', 'log.*'
2020-12-21 21:35:12.001 [info] <0.728.0> Successfully set permissions for 'guest' in virtual host '/' to '.*', '.*', '.*'
2020-12-21 21:35:12.001 [info] <0.730.0> Successfully set permissions for 'guest' in virtual host 'rabbit/hole' to '.*', '.*', '.*'
2020-12-21 21:35:12.001 [info] <0.23994.0> Importing concurrently 1 topic permissions...
2020-12-21 21:35:12.027 [info] <0.726.0> Successfully set topic permissions on exchange 'amq.topic' for 'guest' in virtual host '/' to '.*', '.*'
2020-12-21 21:35:12.027 [info] <0.23994.0> Importing sequentially 1 runtime parameters...
2020-12-21 21:35:12.046 [info] <0.23994.0> Importing sequentially 1 global runtime parameters...
2020-12-21 21:35:12.070 [info] <0.23994.0> Importing concurrently 3 queues...
2020-12-21 21:35:12.071 [info] <0.23994.0> Importing concurrently 4 exchanges...
2020-12-21 21:35:12.071 [info] <0.23994.0> Importing concurrently 4 bindings...

So key information you are looking for is logged at the info level, no need to enable debug logging.

[1993Saurabh]

Thanks @michaelklishin.

Yeah I can see the log but then it throws the error as readiness probe failed.