Webhook communication is always running into a timeout error with a minimal vanilla rke2 installation #5706

catshout · 2024-04-11T09:56:23Z

catshout
Apr 11, 2024

Environmental Info:
RKE2 Version:

rke2 version v1.27.12+rke2r1 (25b27b4e4709a2ac4c550609ad730a9e172d110a)
go version go1.21.8 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

1 Server Node (2 cores, 8 GByte memory, 30 GByte HDD)
Linux srv-sap-eic01 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
1 Agent (Worker) Node (8 Cores, 32 GByte memory, 100 HByte HDD)
Linux srv-sap-eic02 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 server, 1 agent

Describe the bug:

After the rke2 setup I tried to install Longhorn with these commands

helm repo add longhorn https://charts.longhorn.io
helm repo update
helm upgrade -i longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

The Longhorn manager pod shows

W0411 09:45:35.204236       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0411 09:45:35.206428       1 shared_informer.go:311] Waiting for caches to sync for longhorn datastore
I0411 09:45:35.307009       1 shared_informer.go:318] Caches are synced for longhorn datastore
time="2024-04-11T09:45:35Z" level=info msg="Starting longhorn conversion webhook server" func=webhook.StartWebhook file="webhook.go:24"
time="2024-04-11T09:45:35Z" level=info msg="Waiting for conversion webhook to become ready" func=webhook.StartWebhook file="webhook.go:43"
time="2024-04-11T09:45:35Z" level=info msg="Active TLS secret longhorn-system/longhorn-webhook-tls (ver=10644) (count 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=58987BBEAA7E74088A38B87B2FDF0250020BDB18]" func="memory.(*memory).Update" file="memory.go:42"
time="2024-04-11T09:45:35Z" level=info msg="Listening on :9501" func=server.ListenAndServe.func2 file="server.go:77"
time="2024-04-11T09:45:35Z" level=info msg="Started longhorn conversion webhook server on localhost" func=webhook.StartWebhook file="webhook.go:47"
time="2024-04-11T09:45:35Z" level=info msg="Starting /v1, Kind=Secret controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-04-11T09:45:35Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-04-11T09:45:35Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-04-11T09:45:35Z" level=info msg="Building conversion rules..." func="server.(*WebhookServer).runConversionWebhookListenAndServe.func1" file="server.go:195"
time="2024-04-11T09:45:35Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=58987BBEAA7E74088A38B87B2FDF0250020BDB18]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
...
time="2024-04-11T09:46:33Z" level=warning msg="Failed to check endpoint https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz" func=webhook.isServiceAvailable file="webhook.go:78" error="Get \"https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time="2024-04-11T09:46:35Z" level=fatal msg="Error starting manager: conversion webhook service is not accessible after 1m0s sec" func=main.main.DaemonCmd.func3 file="daemon.go:92"

This is one example, similar timeouts messages occur when trying to get rancher up and running.

It seems that the setup described does have a problem in general with the webhook communication.

The Ubuntu firewall is disabled. A NetworkManager doesn't exist.

Steps To Reproduce:

Installed RKE2: following this guide https://docs.rke2.io/install/quickstart
Setup Longhorn with these steps

helm repo add longhorn https://charts.longhorn.io
helm repo update
helm upgrade -i longhorn longhorn/longhorn --namespace longhorn-system --create-namespace

Answered by brandond

Apr 15, 2024

You might try running the ethtool command suggested here?
flannel-io/flannel#1929

View full answer

catshout · 2024-04-11T14:50:08Z

catshout
Apr 11, 2024
Author

Add-On .. I did install Rancher within the rke2 cluster with the following commands, same issue.

# still on  server
# add the cert-manager CRD
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.crds.yaml

# helm install jetstack
helm upgrade -i cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace

# helm install rancher
helm upgrade -i rancher rancher-latest/rancher --create-namespace --namespace cattle-system --set hostname=srv-sap-eic01.domain.com --set bootstrapPassword=bootStrapAllTheThings --set replicas=1

The errors are

2024/04/11 14:46:18 [ERROR] [updateClusterHealth] Failed to update cluster [local]: Internal error occurred: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": context deadline exceeded
2024/04/11 14:46:22 [ERROR] Failed to find system chart rancher-provisioning-capi will try again in 5 seconds: namespaces "cattle-provisioning-capi-system" not found
2024/04/11 14:46:33 [ERROR] [updateClusterHealth] Failed to update cluster [local]: Internal error occurred: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": context deadline exceeded
2024/04/11 14:46:37 [ERROR] Failed to find system chart rancher-provisioning-capi will try again in 5 seconds: namespaces "cattle-provisioning-capi-system" not found

0 replies

brandond · 2024-04-11T15:43:11Z

brandond
Apr 11, 2024
Maintainer

Assuming the pods for the webhooks are actually running, I suspect that something is blocking or dropping CNI traffic between nodes. If you've confirmed that the host's local firewall is disabled, ensure that the correct ports are open on any external firewalls or security groups between your nodes. Ref: https://docs.rke2.io/install/requirements#networking

0 replies

catshout · 2024-04-11T20:51:45Z

catshout
Apr 11, 2024
Author

Tx for the reply. These pods are running momentarily.

kubectl get pods -A -o wide
NAMESPACE                   NAME                                                    READY   STATUS      RESTARTS         AGE   IP              NODE            NOMINATED NODE   READINESS GATES
cattle-fleet-local-system   fleet-agent-6d5df7d9cd-q7ddf                            1/1     Running     0                10h   10.42.1.21      srv-sap-eic02   <none>           <none>
cattle-fleet-system         fleet-controller-877dd5bbc-5nk9h                        1/1     Running     0                10h   10.42.1.14      srv-sap-eic02   <none>           <none>
cattle-fleet-system         gitjob-848764c58b-mkzrw                                 1/1     Running     0                10h   10.42.1.13      srv-sap-eic02   <none>           <none>
cattle-system               rancher-85ff7d6b5-94rk6                                 1/1     Running     0                10h   10.42.1.11      srv-sap-eic02   <none>           <none>
cattle-system               rancher-webhook-869b5d6774-5vlsh                        1/1     Running     0                10h   10.42.1.24      srv-sap-eic02   <none>           <none>
cert-manager                cert-manager-5f8646db6b-rslfs                           1/1     Running     0                10h   10.42.1.9       srv-sap-eic02   <none>           <none>
cert-manager                cert-manager-cainjector-5cf5f57dd7-4bhsm                1/1     Running     0                10h   10.42.1.8       srv-sap-eic02   <none>           <none>
cert-manager                cert-manager-webhook-687b7f8b97-ddzlf                   1/1     Running     0                10h   10.42.1.7       srv-sap-eic02   <none>           <none>
kube-system                 cloud-controller-manager-srv-sap-eic01                  1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 etcd-srv-sap-eic01                                      1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-canal-j5h89                           0/1     Completed   0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-coredns-vrrvb                         0/1     Completed   0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-ingress-nginx-jfk5x                   0/1     Completed   0                12h   10.42.0.4       srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-metrics-server-2mmtz                  0/1     Completed   0                12h   10.42.0.3       srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-snapshot-controller-crd-n79v4         0/1     Completed   0                12h   10.42.0.6       srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-snapshot-controller-xkvs5             0/1     Completed   0                12h   10.42.0.9       srv-sap-eic01   <none>           <none>
kube-system                 helm-install-rke2-snapshot-validation-webhook-r57mn     0/1     Completed   0                12h   10.42.0.5       srv-sap-eic01   <none>           <none>
kube-system                 kube-apiserver-srv-sap-eic01                            1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 kube-controller-manager-srv-sap-eic01                   1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 kube-proxy-srv-sap-eic01                                1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 kube-proxy-srv-sap-eic02                                1/1     Running     0                12h   192.168.40.23   srv-sap-eic02   <none>           <none>
kube-system                 kube-scheduler-srv-sap-eic01                            1/1     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 rke2-canal-b5w6v                                        2/2     Running     0                12h   192.168.40.23   srv-sap-eic02   <none>           <none>
kube-system                 rke2-canal-bzltv                                        2/2     Running     0                12h   192.168.40.22   srv-sap-eic01   <none>           <none>
kube-system                 rke2-coredns-rke2-coredns-864fbd7785-wgq2n              1/1     Running     0                12h   10.42.1.2       srv-sap-eic02   <none>           <none>
kube-system                 rke2-coredns-rke2-coredns-864fbd7785-wl2sh              1/1     Running     0                12h   10.42.0.2       srv-sap-eic01   <none>           <none>
kube-system                 rke2-coredns-rke2-coredns-autoscaler-6c87968579-9msk8   1/1     Running     0                12h   10.42.0.7       srv-sap-eic01   <none>           <none>
kube-system                 rke2-ingress-nginx-controller-jc2wb                     1/1     Running     0                12h   10.42.0.13      srv-sap-eic01   <none>           <none>
kube-system                 rke2-ingress-nginx-controller-l4kl2                     1/1     Running     0                12h   10.42.1.3       srv-sap-eic02   <none>           <none>
kube-system                 rke2-metrics-server-5c9768ff67-fc996                    1/1     Running     0                12h   10.42.0.8       srv-sap-eic01   <none>           <none>
kube-system                 rke2-snapshot-controller-7d6476d7cb-6fkkm               1/1     Running     0                12h   10.42.0.11      srv-sap-eic01   <none>           <none>
kube-system                 rke2-snapshot-validation-webhook-5649fbd66c-t84pn       1/1     Running     0                12h   10.42.0.12      srv-sap-eic01   <none>           <none>
longhorn-system             csi-attacher-57c5fd5bdf-ph6bt                           1/1     Running     0                92m   10.42.0.21      srv-sap-eic01   <none>           <none>
longhorn-system             csi-attacher-57c5fd5bdf-szjrd                           1/1     Running     2 (91m ago)      92m   10.42.1.29      srv-sap-eic02   <none>           <none>
longhorn-system             csi-attacher-57c5fd5bdf-xw7gx                           1/1     Running     2 (91m ago)      92m   10.42.1.28      srv-sap-eic02   <none>           <none>
longhorn-system             csi-provisioner-7b95bf4b87-9tnhq                        1/1     Running     0                92m   10.42.0.19      srv-sap-eic01   <none>           <none>
longhorn-system             csi-provisioner-7b95bf4b87-fgrxj                        1/1     Running     2 (91m ago)      92m   10.42.1.30      srv-sap-eic02   <none>           <none>
longhorn-system             csi-provisioner-7b95bf4b87-qd2qn                        1/1     Running     2 (91m ago)      92m   10.42.1.31      srv-sap-eic02   <none>           <none>
longhorn-system             csi-resizer-6df9886858-2wnkg                            1/1     Running     2 (91m ago)      92m   10.42.1.32      srv-sap-eic02   <none>           <none>
longhorn-system             csi-resizer-6df9886858-7x4rr                            1/1     Running     2 (91m ago)      92m   10.42.1.33      srv-sap-eic02   <none>           <none>
longhorn-system             csi-resizer-6df9886858-rwmrm                            1/1     Running     0                92m   10.42.0.23      srv-sap-eic01   <none>           <none>
longhorn-system             csi-snapshotter-5d84585dd4-dmb76                        1/1     Running     0                92m   10.42.0.20      srv-sap-eic01   <none>           <none>
longhorn-system             csi-snapshotter-5d84585dd4-sg9k4                        1/1     Running     2 (91m ago)      92m   10.42.1.35      srv-sap-eic02   <none>           <none>
longhorn-system             csi-snapshotter-5d84585dd4-xpng8                        1/1     Running     2 (91m ago)      92m   10.42.1.34      srv-sap-eic02   <none>           <none>
longhorn-system             engine-image-ei-5cefaf2b-b6mvl                          1/1     Running     0                10h   10.42.0.17      srv-sap-eic01   <none>           <none>
longhorn-system             engine-image-ei-5cefaf2b-t8pnx                          1/1     Running     0                10h   10.42.1.25      srv-sap-eic02   <none>           <none>
longhorn-system             instance-manager-73a9d47a0bf9b21675bda8dafcf86e76       1/1     Running     0                10h   10.42.0.18      srv-sap-eic01   <none>           <none>
longhorn-system             instance-manager-a0103e2c46498d12462707d90c48f550       1/1     Running     0                94m   10.42.1.26      srv-sap-eic02   <none>           <none>
longhorn-system             longhorn-csi-plugin-dkmvw                               3/3     Running     0                92m   10.42.0.22      srv-sap-eic01   <none>           <none>
longhorn-system             longhorn-csi-plugin-khzb9                               3/3     Running     5 (91m ago)      92m   10.42.1.36      srv-sap-eic02   <none>           <none>
longhorn-system             longhorn-driver-deployer-6f9d655cc8-4cq8t               1/1     Running     4 (93m ago)      11h   10.42.1.5       srv-sap-eic02   <none>           <none>
longhorn-system             longhorn-manager-b22tp                                  1/1     Running     16 (10h ago)     11h   10.42.0.15      srv-sap-eic01   <none>           <none>
longhorn-system             longhorn-manager-hsgk4                                  1/1     Running     104 (100m ago)   11h   10.42.1.6       srv-sap-eic02   <none>           <none>
longhorn-system             longhorn-ui-7cc8f4b459-ffk9r                            1/1     Running     0                11h   10.42.1.4       srv-sap-eic02   <none>           <none>
longhorn-system             longhorn-ui-7cc8f4b459-lk5tc                            1/1     Running     1 (11h ago)      11h   10.42.0.16      srv-sap-eic01   <none>           <none>

I'll go and cross check the network requirements.

0 replies

catshout · 2024-04-11T21:50:33Z

catshout
Apr 11, 2024
Author

Checked the network configuration. Neither Network Manager nor Wicked is running. The server ports 6443 and 9345 are reachable from the agent node.

The kube-apiserver log shows permanently these entries

W0411 21:49:05.881563       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.namespaces.create-non-kubesystem: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded
I0411 21:49:05.881946       1 trace.go:219] Trace[812408867]: "Create" accept:application/json, */*,audit-id:5055d241-bd01-49ca-bb40-b304f185dfe9,client:192.168.40.23,protocol:HTTP/2.0,resource:namespaces,scope:resource,url:/api/v1/namespaces,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:POST (11-Apr-2024 21:48:55.880) (total time: 10001ms):
Trace[812408867]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.namespaces.create-non-kubesystem,resource:/v1, Resource=namespaces,subresource:,operation:CREATE,UID:0976e7f8-278f-48f4-a1ca-52cfcee8ada6 10001ms (21:48:55.880)]
Trace[812408867]: [10.001797085s] [10.001797085s] END

2 replies

brandond Apr 11, 2024
Maintainer

You need a lot more than 6443 and 9345. It is the CNI ports in particular that would handle communication to pods and services running in the cluster, including pod for the webhooks. Refer to the port requirements list I linked previously.

catshout Apr 12, 2024
Author

I've checked the whole setup, all ports between the server and agent (TCP and UDP) are open in general. This I can exclude so far.

catshout · 2024-04-11T21:53:06Z

catshout
Apr 11, 2024
Author

kube-controler-manager shows permanently these entries

E0411 21:45:11.884830       1 garbagecollector.go:392] error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"v1", Kind:"Secret", Name:"helm-operation-lcpr4", UID:"f4710096-1bfc-4fa1-af60-646a44b8d562", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:"cattle-system"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:1}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, dependents:map[*garbagecollector.node]struct {}{}, deletingDependents:false, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, beingDeleted:false, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, virtual:false, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, owners:[]v1.OwnerReference{v1.OwnerReference{APIVersion:"rbac.authorization.k8s.io/v1", Kind:"ClusterRole", Name:"pod-impersonation-helm-op-hjxkj", UID:"cf4e1d2b-4262-4040-8954-0daaadeeeb14", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}}}: Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": context deadline exceeded
E0411 21:45:11.884831       1 garbagecollector.go:392] error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"v1", Kind:"Secret", Name:"pod-impersonation-helm-op-jdbss-token", UID:"9da04776-6b96-42ad-b35e-312aadaa0d1e", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:"cattle-system"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:1}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, dependents:map[*garbagecollector.node]struct {}{}, deletingDependents:false, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, beingDeleted:false, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, virtual:false, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, owners:[]v1.OwnerReference{v1.OwnerReference{APIVersion:"rbac.authorization.k8s.io/v1", Kind:"ClusterRole", Name:"pod-impersonation-helm-op-hjxkj", UID:"cf4e1d2b-4262-4040-8954-0daaadeeeb14", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}}}: Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": context deadline exceeded

0 replies

catshout · 2024-04-15T14:05:15Z

catshout
Apr 15, 2024
Author

@brandond Hi Brad, do you have eventually any other hints for me? Did a lot of research but did find nothing else so far .. Network so far seems to be ok from the checks.

1 reply

brandond Apr 15, 2024
Maintainer

Something is blocking CNI traffic between your nodes. Are all of your nodes on the same network segment? Is there anything else that might be interfering with the traffic?

catshout · 2024-04-15T22:06:24Z

catshout
Apr 15, 2024
Author

Both nodes are in the same segment. The IPs are

ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.40.22  netmask 255.255.255.0  broadcast 192.168.40.255
        inet6 fe80::250:56ff:fe9c:bc13  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9c:bc:13  txqueuelen 1000  (Ethernet)
        RX packets 25888964  bytes 10309348872 (10.3 GB)
        RX errors 0  dropped 149  overruns 0  frame 0
        TX packets 23065083  bytes 10197824290 (10.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::bc55:11ff:fe03:63ba  prefixlen 64  scopeid 0x20<link>
        ether be:55:11:03:63:ba  txqueuelen 0  (Ethernet)
        RX packets 61940  bytes 5494606 (5.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1370574  bytes 82306525 (82.3 MB)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

and

ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.40.23  netmask 255.255.255.0  broadcast 192.168.40.255
        inet6 fe80::250:56ff:fe9c:4e93  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9c:4e:93  txqueuelen 1000  (Ethernet)
        RX packets 30177206  bytes 20440301035 (20.4 GB)
        RX errors 0  dropped 78  overruns 0  frame 0
        TX packets 22267145  bytes 3389843955 (3.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.1.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::5054:2bff:fe08:e8a9  prefixlen 64  scopeid 0x20<link>
        ether 52:54:2b:08:e8:a9  txqueuelen 0  (Ethernet)
        RX packets 1370666  bytes 82312045 (82.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 61947  bytes 5495217 (5.4 MB)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

No idea what else could block the CNI traffic. How could I check this?

2 replies

brandond Apr 15, 2024
Maintainer

You might try running the ethtool command suggested here?
flannel-io/flannel#1929

Answer selected by catshout

catshout Apr 16, 2024
Author

Genial, this did the trick. Both, Rancher and Longhorn now working as expected, mor more Webhook errors! Many thanks Brad @brandond

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Webhook communication is always running into a timeout error with a minimal vanilla rke2 installation #5706

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Webhook communication is always running into a timeout error with a minimal vanilla rke2 installation #5706

catshout Apr 11, 2024

Replies: 7 comments · 5 replies

catshout Apr 11, 2024 Author

brandond Apr 11, 2024 Maintainer

catshout Apr 11, 2024 Author

catshout Apr 11, 2024 Author

brandond Apr 11, 2024 Maintainer

catshout Apr 12, 2024 Author

catshout Apr 11, 2024 Author

catshout Apr 15, 2024 Author

brandond Apr 15, 2024 Maintainer

catshout Apr 15, 2024 Author

brandond Apr 15, 2024 Maintainer

catshout Apr 16, 2024 Author

catshout
Apr 11, 2024

Replies: 7 comments 5 replies

catshout
Apr 11, 2024
Author

brandond
Apr 11, 2024
Maintainer

catshout
Apr 11, 2024
Author

catshout
Apr 11, 2024
Author

brandond Apr 11, 2024
Maintainer

catshout Apr 12, 2024
Author

catshout
Apr 11, 2024
Author

catshout
Apr 15, 2024
Author

brandond Apr 15, 2024
Maintainer

catshout
Apr 15, 2024
Author

brandond Apr 15, 2024
Maintainer

catshout Apr 16, 2024
Author