Rke2 Server won't come up #4122

profhase · 2023-04-13T16:08:08Z

profhase
Apr 13, 2023

Environmental Info:
RKE2 Version:

rke2 version v1.24.9+rke2r2 (2f4571a)
go version go1.18.7b7

Node(s) CPU architecture, OS, and Version:
Linux intern-testkube-bi-01 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 Servers

Describe the bug:
By a deployment one node broke, one by one all nodes disappeared and now no node can come up anymore

On Startup (/usr/bin/rke2 server --debug):

WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.9+rke2r2 (2f4571a879954e1ea8d4560023eaf57c567df737) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
W0413 18:06:36.057943    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2399 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2399: connect: connection refused". Reconnecting...
{"level":"info","ts":"2023-04-13T18:06:36.342+0200","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2400"]}
{"level":"info","ts":"2023-04-13T18:06:36.343+0200","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]}
{"level":"info","ts":"2023-04-13T18:06:36.343+0200","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.18.7b7","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"intern-testkube-bi-01-a9dd0d8f","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-04-13T18:06:36.351+0200","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"8.041877ms"}
{"level":"info","ts":"2023-04-13T18:06:36.970+0200","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":4111326,"snapshot-size":"10 kB"}
{"level":"info","ts":"2023-04-13T18:06:37.537+0200","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":7467008,"backend-size":"7.5 MB","backend-size-in-use-bytes":7458816,"backend-size-in-use":"7.5 MB"}
{"level":"info","ts":"2023-04-13T18:06:37.627+0200","caller":"etcdserver/raft.go:526","msg":"discarding uncommitted WAL entries","entry-index":4115186,"commit-index-from-wal":4115185,"number-of-discarded-entries":49}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"71f2ba444c204ca0","local-member-id":"7924f9bae6efd49b","commit-index":4115186}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b switched to configuration voters=(1172035453513706958 8729376558884574363)"}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b became follower at term 22"}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 7924f9bae6efd49b [peers: [1043e80906d92dce,7924f9bae6efd49b], term: 22, commit: 4115186, applied: 4111326, lastindex: 4115186, lastterm: 22]"}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71f2ba444c204ca0","local-member-id":"7924f9bae6efd49b","recovered-remote-peer-id":"7924f9bae6efd49b","recovered-remote-peer-urls":["https://10.225.13.37:2380"]}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"71f2ba444c204ca0","local-member-id":"7924f9bae6efd49b","recovered-remote-peer-id":"1043e80906d92dce","recovered-remote-peer-urls":["https://10.225.13.47:2380"]}
{"level":"info","ts":"2023-04-13T18:06:37.629+0200","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-04-13T18:06:37.630+0200","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-04-13T18:06:37.631+0200","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":3586614}
{"level":"info","ts":"2023-04-13T18:06:37.639+0200","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":3588961}
{"level":"info","ts":"2023-04-13T18:06:37.640+0200","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-04-13T18:06:37.642+0200","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.642+0200","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.643+0200","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.647+0200","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.647+0200","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce","remote-peer-urls":["https://10.225.13.47:2380"]}
{"level":"info","ts":"2023-04-13T18:06:37.647+0200","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"7924f9bae6efd49b","timeout":"15s"}
{"level":"info","ts":"2023-04-13T18:06:37.649+0200","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.649+0200","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.650+0200","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"warn","ts":"2023-04-13T18:06:37.666+0200","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"7924f9bae6efd49b","requested-revision":3588961,"remote-peer-endpoint":"https://10.225.13.47:2380","error":"Get \"https://10.225.13.47:2380/members/hashkv\": x509: certificate signed by unknown authority"}
{"level":"info","ts":"2023-04-13T18:06:37.666+0200","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"7924f9bae6efd49b"}
{"level":"info","ts":"2023-04-13T18:06:37.666+0200","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"7924f9bae6efd49b","local-server-version":"3.5.4","cluster-id":"71f2ba444c204ca0","cluster-version":"3.5"}
{"level":"info","ts":"2023-04-13T18:06:37.666+0200","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-04-13T18:06:37.668+0200","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"7924f9bae6efd49b","initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-04-13T18:06:37.668+0200","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-04-13T18:06:37.668+0200","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b switched to configuration voters=(8729376558884574363)"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"membership/cluster.go:472","msg":"removed member","cluster-id":"71f2ba444c204ca0","local-member-id":"7924f9bae6efd49b","removed-remote-peer-id":"1043e80906d92dce","removed-remote-peer-urls":["https://10.225.13.47:2380"]}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"7924f9bae6efd49b","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:37.694+0200","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"7924f9bae6efd49b","removed-remote-peer-id":"1043e80906d92dce"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b is starting a new election at term 22"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b became pre-candidate at term 22"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b received MsgPreVoteResp from 7924f9bae6efd49b at term 22"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b became candidate at term 23"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b received MsgVoteResp from 7924f9bae6efd49b at term 23"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"7924f9bae6efd49b became leader at term 23"}
{"level":"info","ts":"2023-04-13T18:06:45.130+0200","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 7924f9bae6efd49b elected leader 7924f9bae6efd49b at term 23"}
{"level":"info","ts":"2023-04-13T18:06:45.131+0200","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"7924f9bae6efd49b","local-member-attributes":"{Name:intern-testkube-bi-01-a9dd0d8f ClientURLs:[http://127.0.0.1:2399]}","request-path":"/0/members/7924f9bae6efd49b/attributes","cluster-id":"71f2ba444c204ca0","publish-timeout":"15s"}
{"level":"info","ts":"2023-04-13T18:06:45.131+0200","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-04-13T18:06:45.131+0200","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0009] Defragmenting etcd database                  
{"level":"info","ts":"2023-04-13T18:06:45.134+0200","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-04-13T18:06:45.135+0200","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":7467008,"current-db-size":"7.5 MB","current-db-size-in-use-bytes":7462912,"current-db-size-in-use":"7.5 MB"}
{"level":"info","ts":"2023-04-13T18:06:45.310+0200","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":0,"current-db-size-bytes":7467008,"current-db-size":"7.5 MB","current-db-size-in-use-bytes-diff":-4096,"current-db-size-in-use-bytes":7458816,"current-db-size-in-use":"7.5 MB","took":"176.636717ms"}
{"level":"info","ts":"2023-04-13T18:06:45.310+0200","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0009] etcd temporary data store connection OK      
INFO[0009] Reconciling bootstrap data between datastore and disk 
DEBU[0009] Reconciling ClientCAKey at '/var/lib/rancher/rke2/server/tls/client-ca.key' 
DEBU[0009] Reconciling ETCDPeerCA at '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' 
DEBU[0009] Reconciling ETCDServerCAKey at '/var/lib/rancher/rke2/server/tls/etcd/server-ca.key' 
DEBU[0009] Reconciling PasswdFile at '/var/lib/rancher/rke2/server/cred/passwd' 
DEBU[0009] Reconciling ClientCA at '/var/lib/rancher/rke2/server/tls/client-ca.crt' 
DEBU[0009] Reconciling ETCDPeerCAKey at '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.key' 
DEBU[0009] Reconciling IPSECKey at '/var/lib/rancher/rke2/server/cred/ipsec.psk' 
DEBU[0009] Reconciling RequestHeaderCA at '/var/lib/rancher/rke2/server/tls/request-header-ca.crt' 
DEBU[0009] Reconciling ServerCA at '/var/lib/rancher/rke2/server/tls/server-ca.crt' 
DEBU[0009] Reconciling ETCDServerCA at '/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' 
DEBU[0009] Reconciling EncryptionHash at '/var/lib/rancher/rke2/server/cred/encryption-state.json' 
DEBU[0009] Reconciling RequestHeaderCAKey at '/var/lib/rancher/rke2/server/tls/request-header-ca.key' 
DEBU[0009] Reconciling ServerCAKey at '/var/lib/rancher/rke2/server/tls/server-ca.key' 
DEBU[0009] Reconciling EncryptionConfig at '/var/lib/rancher/rke2/server/cred/encryption-config.json' 
DEBU[0009] Reconciling ServiceKey at '/var/lib/rancher/rke2/server/tls/service.key' 
INFO[0009] Starting etcd for existing cluster member    
INFO[0009] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=172.17.128.0/17 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key 
INFO[0009] Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 
INFO[0009] Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=172.17.0.0/17 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=172.17.128.0/17 --use-service-account-credentials=true 
INFO[0009] Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/rke2/server/etc/cloud-config.yaml --cloud-provider=rke2 --cluster-cidr=172.17.0.0/17 --configure-cloud-routes=false --controllers=*,-route,-service --kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=rke2-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false 
INFO[0009] Server node token is available at /var/lib/rancher/rke2/server/token 
INFO[0009] To join server node to cluster: rke2 server -s https://10.225.13.37:9345 -t ${SERVER_NODE_TOKEN} 
INFO[0009] Agent node token is available at /var/lib/rancher/rke2/server/agent-token 
INFO[0009] To join agent node to cluster: rke2 agent -s https://10.225.13.37:9345 -t ${AGENT_NODE_TOKEN} 
INFO[0009] Wrote kubeconfig /etc/rancher/rke2/rke2.yaml 
INFO[0009] Run: rke2 kubectl                            
INFO[0009] stopping etcd                                
{"level":"info","ts":"2023-04-13T18:06:45.349+0200","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"intern-testkube-bi-01-a9dd0d8f","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"]}
INFO[0009] Tunnel server egress proxy mode: agent       
INFO[0009] Tunnel server egress proxy waiting for runtime core to become available 
W0413 18:06:45.350410    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/run/k3s/containerd/containerd.sock /run/k3s/containerd/containerd.sock <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory". Reconnecting...
INFO[0009] Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory" 
{"level":"info","ts":"2023-04-13T18:06:45.351+0200","caller":"etcdserver/server.go:1453","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"7924f9bae6efd49b","current-leader-member-id":"7924f9bae6efd49b"}
W0413 18:06:45.351672    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2399 127.0.0.1:2399 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2399: operation was canceled". Reconnecting...
INFO[0009] Cluster-Http-Server 2023/04/13 18:06:45 http: TLS handshake error from 127.0.0.1:44620: remote error: tls: bad certificate 
INFO[0009] Cluster-Http-Server 2023/04/13 18:06:45 http: TLS handshake error from 127.0.0.1:44632: remote error: tls: bad certificate 
{"level":"info","ts":"2023-04-13T18:06:45.400+0200","caller":"embed/etcd.go:563","msg":"stopping serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-04-13T18:06:45.402+0200","caller":"embed/etcd.go:568","msg":"stopped serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-04-13T18:06:45.402+0200","caller":"embed/etcd.go:370","msg":"closed etcd server","name":"intern-testkube-bi-01-a9dd0d8f","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"]}
DEBU[0009] Password verified locally for node 'intern-testkube-bi-01' 
INFO[0009] certificate CN=intern-testkube-bi-01 signed by CN=rke2-server-ca@1680538886: notBefore=2023-04-03 16:21:26 +0000 UTC notAfter=2024-04-12 16:06:45 +0000 UTC 
INFO[0009] certificate CN=system:node:intern-testkube-bi-01,O=system:nodes signed by CN=rke2-client-ca@1680538886: notBefore=2023-04-03 16:21:26 +0000 UTC notAfter=2024-04-12 16:06:45 +0000 UTC 
INFO[0009] Module overlay was already loaded            
INFO[0009] Module nf_conntrack was already loaded       
INFO[0009] Module br_netfilter was already loaded       
INFO[0009] Module iptable_nat was already loaded        
DEBU[0009] getConntrackMax: using conntrack-min         
INFO[0009] Runtime image index.docker.io/rancher/rke2-runtime:v1.24.9-rke2r2 bin and charts directories already exist; skipping extract 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values 
INFO[0009] No cluster configuration value changes necessary for manifest /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values 
INFO[0009] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values 
DEBU[0009] Searching for nvidia container runtime at /usr/local/nvidia/toolkit/nvidia-container-runtime 
DEBU[0009] nvidia container runtime not found at /usr/local/nvidia/toolkit/nvidia-container-runtime 
DEBU[0009] Searching for nvidia container runtime at /usr/bin/nvidia-container-runtime 
DEBU[0009] nvidia container runtime not found at /usr/bin/nvidia-container-runtime 
DEBU[0009] Searching for nvidia-experimental container runtime at /usr/local/nvidia/toolkit/nvidia-container-runtime-experimental 
DEBU[0009] nvidia-experimental container runtime not found at /usr/local/nvidia/toolkit/nvidia-container-runtime-experimental 
DEBU[0009] Searching for nvidia-experimental container runtime at /usr/bin/nvidia-container-runtime-experimental 
DEBU[0009] nvidia-experimental container runtime not found at /usr/bin/nvidia-container-runtime-experimental 
INFO[0009] Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log 
W0413 18:06:45.515207    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/run/k3s/containerd/containerd.sock /run/k3s/containerd/containerd.sock <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory". Reconnecting...
INFO[0009] Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd 
INFO[0010] Containerd is now running                    
DEBU[0010] Deleting existing lease: {rke2 2023-04-13 16:05:50.066070495 +0000 UTC map[]} 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 711.345µs 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 2.259559ms 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 1.956702ms 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 1.723175ms 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt in 308.578µs 
INFO[0010] Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt 
INFO[0010] Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 1.018077ms 
DEBU[0010] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory 
INFO[0010] Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=172.17.128.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=intern-testkube-bi-01 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/rke2 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels= --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key 
INFO[0010] Connecting to proxy                           url="wss://127.0.0.1:9345/v1-rke2/connect"
DEBU[0010] Tunnel server handing HTTP/1.1 GET request for /v1-rke2/connect from 127.0.0.1:44654 
INFO[0010] Handling backend connection request [intern-testkube-bi-01] 
INFO[0010] Cluster-Http-Server 2023/04/13 18:06:46 http: TLS handshake error from 127.0.0.1:44656: remote error: tls: bad certificate 
INFO[0010] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --sync-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --alsologtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cloud-provider has been deprecated, will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.
Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --kubelet-cgroups has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --log-file has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --log-file-max-size has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --stderrthreshold has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
INFO[0014] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0015] Cluster-Http-Server 2023/04/13 18:06:51 http: TLS handshake error from 127.0.0.1:44738: remote error: tls: bad certificate 
DEBU[0015] Wrote ping                                   
INFO[0015] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0019] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0020] Cluster-Http-Server 2023/04/13 18:06:56 http: TLS handshake error from 127.0.0.1:44758: remote error: tls: bad certificate 
DEBU[0020] Wrote ping                                   
INFO[0020] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0024] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0025] Cluster-Http-Server 2023/04/13 18:07:01 http: TLS handshake error from 127.0.0.1:44798: remote error: tls: bad certificate 
DEBU[0025] Wrote ping                                   
INFO[0025] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
W0413 18:07:05.349829    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded". Reconnecting...
W0413 18:07:05.351242    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded". Reconnecting...
INFO[0029] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0029] Container for etcd is running                
INFO[0029] Container for kube-apiserver is running      
INFO[0030] Cluster-Http-Server 2023/04/13 18:07:06 http: TLS handshake error from 127.0.0.1:44820: remote error: tls: bad certificate 
DEBU[0030] Wrote ping                                   
INFO[0030] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0034] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0035] Cluster-Http-Server 2023/04/13 18:07:11 http: TLS handshake error from 127.0.0.1:44884: remote error: tls: bad certificate 
DEBU[0035] Wrote ping                                   
INFO[0035] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0039] Waiting for API server to become available   
{"level":"warn","ts":"2023-04-13T18:07:15.351+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00175a540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\""}
INFO[0039] Failed to test data store connection: context deadline exceeded 
W0413 18:07:15.351276    6372 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
INFO[0039] Waiting for etcd server to become available  
INFO[0039] Tunnel server egress proxy waiting for runtime core to become available 
{"level":"warn","ts":"2023-04-13T18:07:16.582+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00175a540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
ERRO[0040] Failed to check local etcd status for learner management: context deadline exceeded 


```

brandond · 2023-04-13T18:42:46Z

brandond
Apr 13, 2023
Maintainer

Check the static pod logs under /var/log/pods to see why etcd is failing to start. The logs suggest that the etcd pod is running, but etcd is still not available - probably because you no longer have a quorum of servers available? Do you have at least 2/3 servers functional? If not, you might want to run a --cluster-reset to knock the etcd cluster back down to 1 node, then delete and rejoing the remaining servers (assuming you still have an odd number of nodes available)

0 replies

profhase · 2023-04-14T10:58:34Z

profhase
Apr 14, 2023
Author

thank you @brandond , the first server now comes up again after a cluster reset. the others have the same problem ( I pasted the etcd logs below).

Do you know how those things can happen? I would expect the servers come up themselves after a failure

WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.9+rke2r2 (2f4571a879954e1ea8d4560023eaf57c567df737) 
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation. 
INFO[0000] Managed etcd cluster not yet initialized     
WARN[0000] Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation. 
INFO[0000] Reconciling bootstrap data between datastore and disk 
DEBU[0000] Reconciling ClientCA at '/var/lib/rancher/rke2/server/tls/client-ca.crt' 
DEBU[0000] Reconciling ETCDServerCAKey at '/var/lib/rancher/rke2/server/tls/etcd/server-ca.key' 
DEBU[0000] Reconciling ServerCA at '/var/lib/rancher/rke2/server/tls/server-ca.crt' 
DEBU[0000] Reconciling EncryptionConfig at '/var/lib/rancher/rke2/server/cred/encryption-config.json' 
DEBU[0000] Reconciling ServerCAKey at '/var/lib/rancher/rke2/server/tls/server-ca.key' 
DEBU[0000] Reconciling EncryptionHash at '/var/lib/rancher/rke2/server/cred/encryption-state.json' 
DEBU[0000] Reconciling PasswdFile at '/var/lib/rancher/rke2/server/cred/passwd' 
DEBU[0000] Reconciling RequestHeaderCA at '/var/lib/rancher/rke2/server/tls/request-header-ca.crt' 
DEBU[0000] Reconciling ServiceKey at '/var/lib/rancher/rke2/server/tls/service.key' 
DEBU[0000] Reconciling ClientCAKey at '/var/lib/rancher/rke2/server/tls/client-ca.key' 
DEBU[0000] Reconciling ETCDPeerCA at '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' 
DEBU[0000] Reconciling ETCDPeerCAKey at '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.key' 
DEBU[0000] Reconciling ETCDServerCA at '/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' 
DEBU[0000] Reconciling IPSECKey at '/var/lib/rancher/rke2/server/cred/ipsec.psk' 
DEBU[0000] Reconciling RequestHeaderCAKey at '/var/lib/rancher/rke2/server/tls/request-header-ca.key' 
INFO[0000] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=172.17.128.0/17 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key 
INFO[0000] Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 
INFO[0000] Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=172.17.0.0/17 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=172.17.128.0/17 --use-service-account-credentials=true 
INFO[0000] Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/rke2/server/etc/cloud-config.yaml --cloud-provider=rke2 --cluster-cidr=172.17.0.0/17 --configure-cloud-routes=false --controllers=*,-route,-service --kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=rke2-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false
INFO[0000] Server node token is available at /var/lib/rancher/rke2/server/token 
INFO[0000] To join server node to cluster: rke2 server -s https://10.225.13.47:9345 -t ${SERVER_NODE_TOKEN} 
INFO[0000] Agent node token is available at /var/lib/rancher/rke2/server/agent-token 
INFO[0000] To join agent node to cluster: rke2 agent -s https://10.225.13.47:9345 -t ${AGENT_NODE_TOKEN} 
INFO[0000] Wrote kubeconfig /etc/rancher/rke2/rke2.yaml 
INFO[0000] Run: rke2 kubectl                            
W0414 12:49:26.130573   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0000] Tunnel server egress proxy mode: agent       
INFO[0000] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:49:26.130833   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/run/k3s/containerd/containerd.sock /run/k3s/containerd/containerd.sock <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused". Reconnecting...
INFO[0000] Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused" 
W0414 12:49:26.131534   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0000] Cluster-Http-Server 2023/04/14 12:49:26 http: TLS handshake error from 127.0.0.1:46432: remote error: tls: bad certificate 
INFO[0000] Cluster-Http-Server 2023/04/14 12:49:26 http: TLS handshake error from 127.0.0.1:46442: remote error: tls: bad certificate 
DEBU[0000] Password verified locally for node 'intern-testkube-bi-02' 
INFO[0000] certificate CN=intern-testkube-bi-02 signed by CN=rke2-server-ca@1680538886: notBefore=2023-04-03 16:21:26 +0000 UTC notAfter=2024-04-13 10:49:26 +0000 UTC 
INFO[0000] certificate CN=system:node:intern-testkube-bi-02,O=system:nodes signed by CN=rke2-client-ca@1680538886: notBefore=2023-04-03 16:21:26 +0000 UTC notAfter=2024-04-13 10:49:26 +0000 UTC 
INFO[0000] Module overlay was already loaded            
INFO[0000] Module nf_conntrack was already loaded       
INFO[0000] Module br_netfilter was already loaded       
INFO[0000] Module iptable_nat was already loaded        
DEBU[0000] getConntrackMax: using conntrack-min         
INFO[0000] Runtime image index.docker.io/rancher/rke2-runtime:v1.24.9-rke2r2 bin and charts directories already exist; skipping extract 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values 
INFO[0000] No cluster configuration value changes necessary for manifest /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values 
INFO[0000] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values 
DEBU[0000] Searching for nvidia container runtime at /usr/local/nvidia/toolkit/nvidia-container-runtime 
DEBU[0000] nvidia container runtime not found at /usr/local/nvidia/toolkit/nvidia-container-runtime 
DEBU[0000] Searching for nvidia container runtime at /usr/bin/nvidia-container-runtime 
DEBU[0000] nvidia container runtime not found at /usr/bin/nvidia-container-runtime 
DEBU[0000] Searching for nvidia-experimental container runtime at /usr/local/nvidia/toolkit/nvidia-container-runtime-experimental 
DEBU[0000] nvidia-experimental container runtime not found at /usr/local/nvidia/toolkit/nvidia-container-runtime-experimental 
DEBU[0000] Searching for nvidia-experimental container runtime at /usr/bin/nvidia-container-runtime-experimental 
DEBU[0000] nvidia-experimental container runtime not found at /usr/bin/nvidia-container-runtime-experimental 
INFO[0000] Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log 
W0414 12:49:26.325425   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {/run/k3s/containerd/containerd.sock /run/k3s/containerd/containerd.sock <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused". Reconnecting...
INFO[0000] Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd 
W0414 12:49:27.130911   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:27.132172   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0001] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet 
INFO[0002] Containerd is now running                    
DEBU[0002] Deleting existing lease: {rke2 2023-04-14 09:18:57.363928929 +0000 UTC map[]} 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 1.52124ms 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 30.195822ms 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 320.188µs 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 1.077597ms 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt in 7.080235ms 
INFO[0002] Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt 
INFO[0002] Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 288.709µs 
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory 
INFO[0002] Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=172.17.128.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=intern-testkube-bi-02 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/rke2 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels= --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key 
INFO[0002] Connecting to proxy                           url="wss://127.0.0.1:9345/v1-rke2/connect"
W0414 12:49:28.499535   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
DEBU[0002] Tunnel server handing HTTP/1.1 GET request for /v1-rke2/connect from 127.0.0.1:46474 
INFO[0002] Handling backend connection request [intern-testkube-bi-02] 
INFO[0002] Cluster-Http-Server 2023/04/14 12:49:28 http: TLS handshake error from 127.0.0.1:46476: remote error: tls: bad certificate 
INFO[0002] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0002] Adding member intern-testkube-bi-02-4204e571=https://10.225.13.47:2380 to etcd cluster [intern-testkube-bi-01-4baa912e=https://10.225.13.37:2380 intern-testkube-bi-03-20037144=https://10.225.13.48:2380] 
INFO[0002] Starting etcd to join cluster with members [intern-testkube-bi-01-4baa912e=https://10.225.13.37:2380 intern-testkube-bi-03-20037144=https://10.225.13.48:2380 intern-testkube-bi-02-4204e571=https://10.225.13.47:2380] 
Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --sync-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --alsologtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cloud-provider has been deprecated, will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.
Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --kubelet-cgroups has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --log-file has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --log-file-max-size has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --stderrthreshold has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
W0414 12:49:28.869564   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:30.574862   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0005] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:49:31.180914   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0007] Cluster-Http-Server 2023/04/14 12:49:33 http: TLS handshake error from 127.0.0.1:46512: remote error: tls: bad certificate 
INFO[0007] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0007] Wrote ping                                   
W0414 12:49:34.888564   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:35.798442   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0010] Tunnel server egress proxy waiting for runtime core to become available 
DEBU[0012] Wrote ping
INFO[0012] Cluster-Http-Server 2023/04/14 12:49:38 http: TLS handshake error from 127.0.0.1:46532: remote error: tls: bad certificate 
INFO[0012] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0015] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:49:41.536979   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:42.663805   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:43.484310   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0017] Cluster-Http-Server 2023/04/14 12:49:43 http: TLS handshake error from 127.0.0.1:46554: remote error: tls: bad certificate 
INFO[0017] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0017] Wrote ping                                   
W0414 12:49:44.487224   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:46.106841   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0020] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0020] Container for etcd not found (no matching container found), retrying 
W0414 12:49:48.285787   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0022] Cluster-Http-Server 2023/04/14 12:49:48 http: TLS handshake error from 127.0.0.1:46576: remote error: tls: bad certificate 
INFO[0022] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0022] Wrote ping                                   
INFO[0025] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:49:51.928962   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:49:52.303248   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0027] Cluster-Http-Server 2023/04/14 12:49:53 http: TLS handshake error from 127.0.0.1:46596: remote error: tls: bad certificate 
INFO[0027] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0027] Wrote ping                                   
W0414 12:49:54.708578   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
{"level":"warn","ts":"2023-04-14T12:49:56.130+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000900c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
INFO[0030] Failed to test data store connection: context deadline exceeded 
INFO[0030] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0030] Waiting for etcd server to become available  
INFO[0030] Waiting for API server to become available   
{"level":"warn","ts":"2023-04-14T12:49:58.483+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000900c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
ERRO[0032] Failed to check local etcd status for learner management: context deadline exceeded 
W0414 12:49:58.484508   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0032] Cluster-Http-Server 2023/04/14 12:49:58 http: TLS handshake error from 127.0.0.1:46616: remote error: tls: bad certificate 
INFO[0032] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0032] Wrote ping                                   
W0414 12:49:59.484725   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:00.958169   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:01.130927   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0035] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:50:02.132049   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:03.429865   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0037] Cluster-Http-Server 2023/04/14 12:50:03 http: TLS handshake error from 127.0.0.1:46642: remote error: tls: bad certificate 
INFO[0037] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0037] Wrote ping                                   
W0414 12:50:03.599796   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0040] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0040] Container for etcd not found (no matching container found), retrying 
W0414 12:50:06.496099   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:08.124474   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0042] Cluster-Http-Server 2023/04/14 12:50:08 http: TLS handshake error from 127.0.0.1:46664: remote error: tls: bad certificate 
INFO[0042] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0042] Wrote ping                                   
INFO[0045] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:50:11.251831   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:11.715210   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
{"level":"warn","ts":"2023-04-14T12:50:13.484+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000900c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
ERRO[0047] Failed to check local etcd status for learner management: context deadline exceeded 
W0414 12:50:13.485534   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0047] Cluster-Http-Server 2023/04/14 12:50:13 http: TLS handshake error from 127.0.0.1:46686: remote error: tls: bad certificate 
INFO[0047] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0047] Wrote ping                                   
W0414 12:50:14.485941   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
W0414 12:50:16.023568   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0050] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:50:17.219822   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0052] Cluster-Http-Server 2023/04/14 12:50:18 http: TLS handshake error from 127.0.0.1:46708: remote error: tls: bad certificate 
W0414 12:50:18.505165   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0052] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0052] Wrote ping                                   
INFO[0055] Tunnel server egress proxy waiting for runtime core to become available 
W0414 12:50:21.835905   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
INFO[0057] Cluster-Http-Server 2023/04/14 12:50:23 http: TLS handshake error from 127.0.0.1:46728: remote error: tls: bad certificate 
INFO[0057] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
DEBU[0057] Wrote ping                                   
INFO[0060] Waiting for API server to become available   
INFO[0060] Waiting for etcd server to become available  
INFO[0060] Tunnel server egress proxy waiting for runtime core to become available 
INFO[0060] Container for etcd not found (no matching container found), retrying 
W0414 12:50:27.228220   10443 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
{"level":"warn","ts":"2023-04-14T12:50:28.490+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000900c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}
ERRO[0062] Failed to check local etcd status for learner management: context deadline exceeded

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rke2 Server won't come up #4122

{{title}}

Replies: 2 comments

{{title}}

{{title}}

Select a reply

Rke2 Server won't come up #4122

profhase Apr 13, 2023

Replies: 2 comments

brandond Apr 13, 2023 Maintainer

profhase Apr 14, 2023 Author

profhase
Apr 13, 2023

brandond
Apr 13, 2023
Maintainer

profhase
Apr 14, 2023
Author