shutdown RKE2 cluster

[mcguppy]

We have the use case to be able to shutdown an RKE2 cluster (all masters and all workers) in case of an power outage during the time the emergency power from batterie is still available (about 10 minutes).

What is the correct procedure to shutdown a whole cluster?
I tried to do a systemctl stop rke2-server rke2-agent first on all workers and after about 30 sec to do a shutdown -h now on all workers. Then I did the same on all master nodes.
All the master nodes shut down correctly, but the workers get stucked in shutting down.
After about 30 minutes, I did a hard power down on all workers.
When switching on all again, the cluster comes back online and all apps as well.
But I need a solution to really shutdown all workers the correct way, so I don't have to do it the hard way.

Is there any correct procedure for this? I have searched the web for quite a long time and the only doc I found was https://www.suse.com/support/kb/doc/?id=000020031
This is not possible as I do not have docker as container engine but containerd.

Thank you very much for any help on this.

[briandowns]

That seems like a reasonable procedure. When you say that the workers are hung, is it the RKE2 process that's hanging? If so, what is in the logs at the time of the shutdown/hang?

[Martin-Weiss]

@kkmaslowski - how did you deploy the cluster? Do you have a load balancer VIP for server= / for port 9345 and 6443?

[kkmaslowski]

@Martin-Weiss setup is almost the same as described here: https://computingforgeeks.com/deploy-kubernetes-on-rocky-using-rke2/
3 servers, 2 agents, load balancer. Now I see one difference - on second and third server in file /etc/rancher/rke2/config.yaml I have "server: https://server1.example.com:9345". I should set here address of balancer, not first server.The same for agents.

Please clarify - after installing all servers should I add 'server:' and 'token' also on first server? Both options should be added on all servers?

Server that failed in my setup is server1. It had no 'server:' and 'token:' options, it was installed as first.

[Martin-Weiss]

@kkmaslowski - yes - server: should go to the load balancer for :9345 (which goes to the three masters) and after installation you should add the server: also on the first one.

[kkmaslowski]

@Martin-Weiss thanks for answers. We will reconfigure it as you described.

[bovy89]

can you provide more details about that here? #3701