Skip to content

Commit

Permalink
Update rabbitmq-reset for Epoxy
Browse files Browse the repository at this point in the history
With the introduction of oslo.messaging Queue Manager, we need to clear
shared memory when wiping queues from RabbitMQ. This is because
oslo.messaging assumes that a queue with an entry under shared memory
exists in the RabbitMQ cluster.

With the introduction of durable queues for reply/fanout queues, we need
stop the services before wiping queues from RabbitMQ. This is because
fanout queues in particular are not just created on container startup,
so services may attempt to use them after the RabbitMQ state has been
reset.

Removes the tags as it no longer makes sense to only run part of this
playbook.
  • Loading branch information
MoteHue committed Feb 4, 2025
1 parent 77ea637 commit 922b044
Showing 1 changed file with 22 additions and 6 deletions.
28 changes: 22 additions & 6 deletions etc/kayobe/ansible/rabbitmq-reset.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,25 @@
# Reset a broken RabbitMQ cluster.
# Also restarts all OpenStack services using RabbitMQ.

- name: Stop OpenStack services
hosts: controllers:compute:storage
become: true
gather_facts: false
tasks:
# The following services use RabbitMQ.
- name: Restart OpenStack services
ansible.builtin.shell:
cmd: >-
set -o pipefail &&
systemctl -a | egrep 'kolla-(barbican|blazar|cinder|cloudkitty|designate|heat|ironic|keystone|magnum|manila|neutron|nova|octavia)' |
awk '{ print $NF }' |
xargs systemctl stop
executable: "/bin/bash"

- name: Reset RabbitMQ
hosts: controllers
become: true
gather_facts: false
tags:
- rabbitmq-reset
vars:
container_name: rabbitmq
tasks:
Expand Down Expand Up @@ -59,18 +72,21 @@
ansible.builtin.command: docker exec {{ container_name }} /bin/bash -c 'rabbitmqctl await_online_nodes {{ groups['controllers'] | length }}'

- name: Restart OpenStack services
hosts: controllers:compute
hosts: controllers:compute:storage
become: true
gather_facts: false
tags:
- restart-openstack
tasks:
- name: Clear shared memory
ansible.builtin.shell:
cmd: >
rm -rf /dev/shm/*_qmanager
# The following services use RabbitMQ.
- name: Restart OpenStack services
ansible.builtin.shell:
cmd: >-
set -o pipefail &&
systemctl -a | egrep 'kolla-(barbican|blazar|cinder|cloudkitty|designate|heat|ironic|keystone|magnum|manila|neutron|nova|octavia)' |
awk '{ print $1 }' |
awk '{ print $NF }' |
xargs systemctl restart
executable: "/bin/bash"

0 comments on commit 922b044

Please sign in to comment.