Skip to content
This repository has been archived by the owner on Apr 25, 2019. It is now read-only.

API server fails to start (0.9.1) #26

Open
cmoulliard opened this issue Jan 24, 2019 · 6 comments
Open

API server fails to start (0.9.1) #26

cmoulliard opened this issue Jan 24, 2019 · 6 comments

Comments

@cmoulliard
Copy link

cmoulliard commented Jan 24, 2019 •
edited
Loading

I have been able on my macbook machine to create the 2 virtualbox vms using ./minishift_virtualbox.sh create && ./minishift_virtualbox.sh start but when I try to log on to the API server, it fails

oc login https://192.168.99.100:6443 -u system:admin
error: dial tcp 192.168.99.100:6443: connect: connection refused - verify you have provided the correct host and port and that the server is currently running.

If I ssh to the master vm and checks the kubelet service, then such connection erros occur

ssh [email protected] -p 5222
[email protected]'s password:
Last login: Thu Jan 24 15:38:22 2019 from gateway
Red Hat CoreOS 4.0
 Information: https://url.corp.redhat.com/redhat-coreos
 Bugs: https://github.com/openshift/os

---
[core@test1-master-0 ~]$ systemctl status kubelet.service
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-01-24 16:46:30 UTC; 59min left
  Process: 2877 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 2886 (hyperkube)
   Memory: 118.0M
   CGroup: /system.slice/kubelet.service
           └─2886 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig -...

Jan 24 15:51:54 test1-master-0 hyperkube[2886]: E0124 15:51:54.009137    2886 kubelet_node_status.go:103] Unable to register node "test1-master-0" with API server: Post https://test1-api.tt.testing:6443/api/v1/nodes: dial tcp 192.168.99.100:6443: connect: connection refused
...
Jan 24 15:46:46 test1-master-0 hyperkube[2886]: E0124 15:46:46.477546    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.489664    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.489672    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.490960    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499286    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499286    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499955    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.510005    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.510102    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.511426    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused

Port opened

[root@test1-master-0 core]# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2886/hyperkube
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:46581           0.0.0.0:*               LISTEN      3045/rpc.statd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2936/sshd
tcp        0      0 10.0.3.15:10010         0.0.0.0:*               LISTEN      3040/crio
tcp6       0      0 :::10250                :::*                    LISTEN      2886/hyperkube
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      2936/sshd
tcp6       0      0 :::59395                :::*                    LISTEN      3045/rpc.statd
@gbraad
Copy link
Collaborator

gbraad commented Jan 24, 2019

The port is not opened and it clearly shows issues with the API server.
Have you restarted?

@cmoulliard
Copy link
Author

If I do ./minishift_virtualbox.sh stop followed by ./minishift_virtualbox.sh start, then the same error is reported

ssh [email protected]
[email protected]'s password:
Last login: Thu Jan 24 15:57:06 2019 from 192.168.99.1
Red Hat CoreOS 4.0
 Information: https://url.corp.redhat.com/redhat-coreos
 Bugs: https://github.com/openshift/os

---
[core@test1-master-0 ~]$ sudo lsof -i -P -n | grep LISTEN
sudo: lsof: command not found
[core@test1-master-0 ~]$ sudo su
[root@test1-master-0 core]# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2899/hyperkube
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:39990           0.0.0.0:*               LISTEN      3033/rpc.statd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2938/sshd
tcp        0      0 10.0.3.15:10010         0.0.0.0:*               LISTEN      3030/crio
tcp6       0      0 :::10250                :::*                    LISTEN      2899/hyperkube
tcp6       0      0 :::60906                :::*                    LISTEN      3033/rpc.statd
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      2938/sshd

and

Jan 24 16:11:51 test1-master-0 hyperkube[2899]: E0124 16:11:51.892882    2899 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dtest1-master-0&limit=500&resourceVersion=0: dial tcp 192.168.99.100:6443: connect: connection refused

Why kubelet is not able to access the API server ?

@cmoulliard
Copy link
Author

Here is what crictl reports

[root@test1-master-0 core]# crictl images
IMAGE                                                 TAG                                        IMAGE ID            SIZE
k8s.gcr.io/pause                                      3.1                                        da86e6ba6ca19       747kB
quay.io/coreos/etcd                                   v3.3.10                                    643c21638c1c9       39.7MB
quay.io/coreos/kube-client-agent                      36c62ccd7b16b522450c61e96fc556b217ee24f5   b02de22ff740f       33.8MB
quay.io/openshift-release-dev/ocp-release             <none>                                     d0bc83da1db0c       292MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1484c1de242d8       316MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     eeee805d24429       576MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     007942e83b562       288MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e0972dab0d058       285MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     430af544425fa       374MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     73f1f98cd6dd9       316MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     169a72c94dcde       281MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     9d15e9bac6339       366MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     c779d02a634cc       315MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e243d7975e99c       285MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     bfa0ad6d2753f       295MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     d56d47ed50c2f       374MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e7ebbc9e5eb03       514MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     edca0ee739703       314MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     0ee50df593f6d       312MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     7f8e4a1588a21       268MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1c9c9fa847ef1       293MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     66499234b190f       299MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     b0d2d081a1eeb       287MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     090cff599214a       288MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     0b374c5262383       294MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     40583808760aa       317MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1319a581abe9a       311MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     2e30054dab0ab       322MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     f5a02bf81b079       290MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     459d56445a917       1.25GB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     b867876bda666       278MB
registry.svc.ci.openshift.org/openshift/origin-v4.0   setup-etcd-environment                     ef87ffbfa30dc       248MB
[root@test1-master-0 core]# crictl pods
POD ID              CREATED             STATE               NAME                                                              NAMESPACE                                                 ATTEMPT
dd180d0bbbf4a       12 minutes ago      NotReady            etcd-member-test1-master-0                                        kube-system                                               1
27c1ec3ad57cf       12 minutes ago      NotReady            openshift-kube-apiserver-test1-master-0                           openshift-kube-apiserver                                  1
eb2eb97c751a3       12 minutes ago      NotReady            openshift-kube-controller-manager-test1-master-0                  openshift-kube-controller-manager                         1
7981a2cc5f19f       12 minutes ago      NotReady            openshift-kube-scheduler-test1-master-0                           openshift-kube-scheduler                                  1
2bfcea9198333       About an hour ago   NotReady            cluster-autoscaler-operator-6855f55d94-29jkx                      openshift-cluster-api                                     1
9bdcfa7ed3115       About an hour ago   NotReady            dns-default-9kcsf                                                 openshift-dns                                             1
c96bde52fc5e9       About an hour ago   NotReady            machine-config-operator-769967ddf5-sspjj                          openshift-machine-config-operator                         1
fd0e773926d5e       About an hour ago   NotReady            origin-cluster-osin-operator-774dc44fd8-rs76w                     openshift-core-operators                                  1
a879cd9573fae       About an hour ago   NotReady            openshift-console-85d4d65d64-mbgpk                                openshift-console                                         1
7bcee689097c1       About an hour ago   NotReady            openshift-service-cert-signer-operator-784d9677b5-vlr2m           openshift-core-operators                                  1
b53a20cece153       About an hour ago   NotReady            machine-api-operator-67f964b4d-tdnk5                              openshift-cluster-api                                     1
fe43242899947       About an hour ago   NotReady            configmap-cabundle-injector-755f877875-wv886                      openshift-service-cert-signer                             1
6e0787fe49312       About an hour ago   NotReady            openshift-cluster-openshift-controller-manager-operator-5dwhnht   openshift-cluster-openshift-controller-manager-operator   1
7a7b18a366ab1       About an hour ago   NotReady            openshift-kube-controller-manager-operator-66cbb459f8-6n6dg       openshift-kube-controller-manager-operator                1
7c60c5fa3881c       About an hour ago   NotReady            origin-cluster-osin-operator2-669747b677-txl8z                    openshift-core-operators                                  1
c6121c5b5ddbe       About an hour ago   NotReady            dns-operator-6dbdd7df84-lvm86                                     openshift-dns-operator                                    1
fa7bfba3ee485       About an hour ago   NotReady            openshift-console-85d4d65d64-qbppk                                openshift-console                                         1
b48256d4085c7       About an hour ago   NotReady            openshift-cluster-kube-scheduler-operator-7776d4f85c-2s7t6        openshift-cluster-kube-scheduler-operator                 1
91b89b8555075       About an hour ago   NotReady            image-registry-6fb8cbfb4f-q4sqt                                   openshift-image-registry                                  1
5a6da8e3d57c7       About an hour ago   NotReady            openshift-console-85d4d65d64-k8bbt                                openshift-console                                         1
afcebd9899048       About an hour ago   NotReady            cluster-samples-operator-7cf8d8cf66-svddq                         openshift-cluster-samples-operator                        1
6c73717a1259f       About an hour ago   NotReady            apiservice-cabundle-injector-7cc897d85b-pbwh5                     openshift-service-cert-signer                             1
a1af578009df6       About an hour ago   NotReady            clusterapi-manager-controllers-748cb8d66c-68c8w                   openshift-cluster-api                                     1
ef4f10b5a8586       About an hour ago   NotReady            registry-ca-hostmapper-w6l68                                      openshift-image-registry                                  1
4bc7fd9f6280d       About an hour ago   NotReady            openshift-kube-apiserver-operator-58b8c455c5-4jhqg                openshift-kube-apiserver-operator                         1
dc63b9ad5b246       About an hour ago   NotReady            apiserver-sf68z                                                   openshift-apiserver                                       1
1a5a3d2320d77       About an hour ago   NotReady            machine-config-controller-68f5989588-p45v9                        openshift-machine-config-operator                         1
77dc3704f4e05       About an hour ago   NotReady            service-serving-cert-signer-66f9694c4d-8l878                      openshift-service-cert-signer                             1
8c574c82313ce       About an hour ago   NotReady            openshift-apiserver-operator-77756dfc66-942df                     openshift-apiserver-operator                              1
d6d32c2801713       About an hour ago   NotReady            controller-manager-6qp2c                                          openshift-controller-manager                              1
d265f3412b6c4       About an hour ago   NotReady            cluster-image-registry-operator-6c4478c54c-kb7rh                  openshift-image-registry                                  1
965abe7e454a9       About an hour ago   NotReady            tuned-z77hj                                                       openshift-cluster-node-tuning-operator                    1
2b6590f633081       About an hour ago   NotReady            machine-config-server-jp594                                       openshift-machine-config-operator                         1
4c9eead87f8a8       About an hour ago   NotReady            machine-config-daemon-s4xxc                                       openshift-machine-config-operator                         1
937d3ebb7ecf9       About an hour ago   NotReady            machine-approver-86b68b66f7-wvztd                                 openshift-cluster-machine-approver                        1
d02712f26ecd2       About an hour ago   NotReady            cluster-network-operator-vx7xr                                    openshift-cluster-network-operator                        1
297926230d315       About an hour ago   NotReady            sdn-nzdgx                                                         openshift-sdn                                             1
9868e0dede288       About an hour ago   NotReady            sdn-controller-6svcg                                              openshift-sdn                                             1
aa0860f318bb7       About an hour ago   NotReady            cluster-version-operator-7b47d58bff-j6nb7                         openshift-cluster-version                                 1
944f8b6783bc8       About an hour ago   NotReady            ovs-xfdg6                                                         openshift-sdn                                             1
4bb64351080a3       2 weeks ago         NotReady            installer-3-test1-master-0                                        openshift-kube-apiserver                                  0
3b1c8e1f2d4f6       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-controller-manager                         0
20bcea5fc476e       2 weeks ago         NotReady            installer-2-test1-master-0                                        openshift-kube-apiserver                                  0
791d3200beeb8       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-apiserver                                  0
28629cdfda637       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-scheduler                                  0
[root@test1-master-0 core]#

and

[root@test1-master-0 core]# crictl inspectp 27c1ec3ad57cf
{
  "status": {
    "id": "27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84",
    "metadata": {
      "attempt": 1,
      "name": "openshift-kube-apiserver-test1-master-0",
      "namespace": "openshift-kube-apiserver",
      "uid": "8f63e2613f94443f7263843d3501805f"
    },
    "state": "SANDBOX_NOTREADY",
    "createdAt": "2019-01-24T16:01:25.232664761Z",
    "network": {
      "ip": ""
    },
    "linux": {
      "namespaces": {
        "options": {
          "ipc": "POD",
          "network": "NODE",
          "pid": "CONTAINER"
        }
      }
    },
    "labels": {
      "apiserver": "true",
      "app": "openshift-kube-apiserver",
      "io.kubernetes.container.name": "POD",
      "io.kubernetes.pod.name": "openshift-kube-apiserver-test1-master-0",
      "io.kubernetes.pod.namespace": "openshift-kube-apiserver",
      "io.kubernetes.pod.uid": "8f63e2613f94443f7263843d3501805f",
      "revision": "3"
    },
    "annotations": {
      "kubernetes.io/config.hash": "8f63e2613f94443f7263843d3501805f",
      "kubernetes.io/config.seen": "2019-01-24T16:01:24.098343869Z",
      "kubernetes.io/config.source": "file"
    }
  },
  "version": {
    "version": "1.11.10"
  }
}
[root@test1-master-0 core]# crictl logs 27c1ec3ad57cf
E0124 16:15:35.331505    3977 remote_runtime.go:278] ContainerStatus "27c1ec3ad57cf" from runtime service failed: rpc error: code = Unknown desc = specified container not found: 27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84
FATA[0000] rpc error: code = Unknown desc = specified container not found: 27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84
[root@test1-master-0 core]#

@gbraad
Copy link
Collaborator

gbraad commented Jan 24, 2019

27c1ec3ad57cf 12 minutes ago NotReady openshift-kube-apiserver-test1-master-0

Not working as expected. Let's disuss this tomorrow, as @praveenkumar and I need to also verify this. It might be related to pruning.

@gbraad gbraad changed the title Can't log using oc to the API Server API server fails to start (0.9.1) Jan 24, 2019
@cmoulliard
Copy link
Author

If I try to create a pod then it fails with this error

[root@test1-master-0 core]# crictl runp pod-config.json
FATA[0000] run pod sandbox failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_nginx-sandbox_default_hdishd83djaidwnduwk28bcsb_1(81dbaeafd1d872b88debd40df631d786d2642aff6653a94403b322deda046fa7): OpenShift SDN network process is not (yet?) available

@gbraad
Copy link
Collaborator

gbraad commented Jan 24, 2019

OpenShift SDN will not start properly due to #17

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gbraad @cmoulliard