Question about strange behaviour

[matsbror]

I am setting up a heterogeneous cluster with a number of ARM64 nodes, one RISCV64 node and one or more x86-64-nodes.

I first tried with one of the ARM-nodes as the controller. I could set up the other nodes, but I could not get the pod to start on the riscv node (I am using the hello-kubernetes example and using nodeSelector: kubernetes.io/arch: riscv64).
The pod was assigned to the right node but I got the following error which I could see in journalctl:

kubelet Error: services have not yet been read at least once, cannot construct envvars

When I used arm64 as desired node, it worked fine. The ARM nodes were installed from the official k3s page.

I then switch roles and put the controller on the riscv node and made the ARM nodes agents. With this configuration I could run hello-kubernetes fine both on ARM nodes as well as on the RISCV node.

I can live with this, but I'd rather have the RISCV node as an agent.

Any ideas on why I have experienced this behaviour?

[chazapis]

Hi! It seems that this error usually shows up when the underlying network has issues. Have you changed anything at the networking side when changing the controller architecture? What CNI are you using? Are all nodes running K3s?

[matsbror]

Nothing was changed on the network side and all nodes are running k3s. The arm nodes directly from the k3s.io site and the riscv node from this repo.

I am new to Kubernetes so I do not yet know what a CNI is and I certainly have not configured one. I just used the getting started instructions on k3s.io.

[chazapis]

Ok. I have made a note to try and duplicate this on my side. I will do and get back to you when I have some progress.

[matsbror]

I had my IT-people to make sure that the traffic between my nodes is open on all ports.

I replicated the issue but with an x86 machine as the controller. I think I will need to revert having the riscv node as controller.

[matsbror]

An update: I tried again with one of the ARM nodes as the control node, and now it works as expected. I can start pods on the riscv64 node using:

nodeSelector:
        kubernetes.io/arch: riscv64

It got assigned to the right node and finished execution correctly.

Still do not know what the issue is.

[chazapis]

Thanks for the heads up!