rke2-agent node is not able to join to rke2-server cluster due to error: failed to get CA certs

[anishagoyal0504]

Hi Team,
My rke2-server is working fine and can see the rke2-server node in the cluster (kubectl get nodes).
Now I am trying to join rke2-agent node in the rke2-server cluster by starting the service however, it is failing on the below:

"failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

my rke2-agent node has below config to join the rke2-server clsuter:

server: https://<rke2-server private ip address>:9345
token: <value of node-token>

Am I missing any steps for installing rke2-server and rke-agent. I am using standard method for installation (curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -)

Any clue, why this error is coming, I could not find any solid answer or solution either on google or rke2- troubleshooting guides.

Any view please

thanks

[brandond]

It looks like the agent can't connect to the server? The connection is timing out. Make sure that you can successfully curl the server address from the agent.

[anishagoyal0504]

I found the issue on AWS linux VM Inbound security rules on RKE2-Server, when I do netstat on rke2-server, it shows the ports are opening on tcp6 (::/0) IPv6, when I added IPv6 (::/0) in inbound security rules, rke2-agent node was able to join the rke2 cluster. I was under the impression that particularly these ports (6443, 9345) are should be running under TCP4 on rke2-server. you might need to add this in RKE2 troubleshooting guidelines so that people knows straightway.

[anishagoyal0504]

Hi @brandond I have created another rke2 cluster on aws EC2 instance. I am having same issue with joining rke2-agent to rke2-server, when I start the rke2-agent service it gives me the error:
failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": context deadline exceeded
why this is misleading logs and why it is pointing to port 6444, what to do with this port?

in my rke2-server config.yaml has:
server: https://10.119.61.128:9345
token: K10401aeba15555fd9c124f50f516fe9399b41af273a93469d68077e716d5351de6::server:075ba65d8e7d20e07ecd5555cbf87f25

Can you please me with this and there is known issue on AWS. When I run the same setup on my local desktop machine instead of AWS, it works fine

[manuelbuil]

6444 is the proxy port. It just makes sense locally. That will forward the connection to the server, if you check the logs, you'll see something like this:

 rke2[45304]: time="2024-08-16T11:28:55+10:00" level=info msg="Running load balancer rke2-agent-load-balancer 127.0.0.1:6444 -> $IP:$PORT

Please make sure your AWS security group has $PORT open

[anishagoyal0504]

Hi @manuelbuil , my aws security group has all the ports (9345,6443,2379,10250, 6444) are open in inbound security rules. I have no idea why it is still causing this issue?

running netstat on rke2-server:
[ec2-user@ip-10-119-61-128 ~]$ netstat -aln | grep 9345
tcp 0 0 10.119.61.128:40988 10.119.61.128:9345 ESTABLISHED
tcp 0 0 127.0.0.1:37442 127.0.0.1:9345 ESTABLISHED
tcp6 0 0 :::9345 :::* LISTEN
tcp6 0 0 127.0.0.1:9345 127.0.0.1:37442 ESTABLISHED
tcp6 0 0 10.119.61.128:9345 10.119.61.128:40988 ESTABLISHED

when I trying to curl from rke2-agent:
curl -vks https://10.119.61.128:9345/cacerts

nothing happened

looking at logs of rke2-agent node it says:
"failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": context deadline ex

It looks like it is known issue on AWS based VM, it should be documented in your rancher guidlines

Can you help please

[brandond]

If you can't curl the server URL then there is some other issue at the network level. There are no known issues with running rke2 on EC2. You just need to make sure that the nodes can reach each other, same as in any other environment.