MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found

[mpaguilar]

I get the "502 endpoint not found" error when trying to setup the datastore.

A little investigation found this error in the controller pod:
MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found

This is a k3s installation, pretty fresh except for a couple of web-serving pods.

Here's more detail:

$ kubectl version
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.4+k3s1

$ kubectl apply --server-side -k "https://github.com/piraeusdatastore/piraeus-operator//config/default?ref=v2"
namespace/piraeus-datastore serverside-applied
customresourcedefinition.apiextensions.k8s.io/linstorclusters.piraeus.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/linstornodeconnections.piraeus.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/linstorsatelliteconfigurations.piraeus.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/linstorsatellites.piraeus.io serverside-applied
serviceaccount/piraeus-operator-controller-manager serverside-applied
serviceaccount/piraeus-operator-gencert serverside-applied
role.rbac.authorization.k8s.io/piraeus-operator-gencert serverside-applied
role.rbac.authorization.k8s.io/piraeus-operator-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/piraeus-operator-controller-manager serverside-applied
clusterrole.rbac.authorization.k8s.io/piraeus-operator-gencert serverside-applied
rolebinding.rbac.authorization.k8s.io/piraeus-operator-gencert serverside-applied
rolebinding.rbac.authorization.k8s.io/piraeus-operator-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/piraeus-operator-gencert serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/piraeus-operator-manager-rolebinding serverside-applied
configmap/piraeus-operator-image-config serverside-applied
service/piraeus-operator-webhook-service serverside-applied
deployment.apps/piraeus-operator-controller-manager serverside-applied
deployment.apps/piraeus-operator-gencert serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/piraeus-operator-validating-webhook-configuration serverside-applied

$ kubectl -n piraeus-datastore get pods
NAME                                                   READY   STATUS    RESTARTS   AGE
piraeus-operator-controller-manager-6c9bf78dfc-k9ncs   1/1     Running   0          60s
piraeus-operator-gencert-9ddfd549b-p7ncn               1/1     Running   0          60s

$ kubectl describe -n piraeus-datastore pod piraeus-operator-controller-manager-6c9bf78dfc-k9ncs
Name:                 piraeus-operator-controller-manager-6c9bf78dfc-k9ncs
Namespace:            piraeus-datastore
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      piraeus-operator-controller-manager
Node:                 middlebox/10.1.40.10
Start Time:           Fri, 17 Jan 2025 08:24:04 -0600
Labels:               app.kubernetes.io/component=piraeus-operator
                      app.kubernetes.io/name=piraeus-datastore
                      pod-template-hash=6c9bf78dfc
Annotations:          kubectl.kubernetes.io/default-container: manager
Status:               Running
IP:                   10.42.3.14
IPs:
  IP:           10.42.3.14
Controlled By:  ReplicaSet/piraeus-operator-controller-manager-6c9bf78dfc
Containers:
  manager:
    Container ID:  containerd://3bb12ea69861d0226b4e7f563aa02395b36f0668669cfd4d5d39626b5c796248
    Image:         quay.io/piraeusdatastore/piraeus-operator:v2
    Image ID:      quay.io/piraeusdatastore/piraeus-operator@sha256:0ecba47d7cdd6444c06845a9c0f83ff7a4a8d51f91c01856fde14874c8f2be5f
    Port:          9443/TCP
    Host Port:     0/TCP
    Command:
      /manager
    Args:
      --leader-elect
      --metrics-bind-address=0
      --namespace=$(NAMESPACE)
      --image-config-map-name=$(IMAGE_CONFIG_MAP_NAME)
    State:          Running
      Started:      Fri, 17 Jan 2025 08:24:36 -0600
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  256Mi
    Requests:
      cpu:      10m
      memory:   64Mi
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      NAMESPACE:              piraeus-datastore (v1:metadata.namespace)
      IMAGE_CONFIG_MAP_NAME:  piraeus-operator-image-config
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rptzv (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webhook-server-cert
    Optional:    false
  kube-api-access-rptzv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 drbd.linbit.com/force-io-error:NoSchedule
                             drbd.linbit.com/lost-quorum:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    2m41s                  default-scheduler  Successfully assigned piraeus-datastore/piraeus-operator-controller-manager-6c9bf78dfc-k9ncs to middlebox
  Warning  FailedMount  2m25s (x6 over 2m41s)  kubelet            MountVolume.SetUp failed for volume "cert" : secret "webhook-server-cert" not found
  Normal   Pulled       2m9s                   kubelet            Container image "quay.io/piraeusdatastore/piraeus-operator:v2" already present on machine
  Normal   Created      2m9s                   kubelet            Created container manager
  Normal   Started      2m9s                   kubelet            Started container manager

Any help is appreciated. Thank you.

[WanzenBug]

Looks like everything is running? The warning happens when you initially deploy the operator, as the gencert container tykes some time to create the certficates. I'm not sure what this is supposed to mean:

I get the "502 endpoint not found" error when trying to setup the datastore.

Can you elaborate what the actual issue is?

[mpaguilar]

$ kubectl apply -f - <<EOF
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec: {}
EOF
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "vlinstorcluster.kb.io": failed to call webhook: Post "https://piraeus-operator-webhook-service.piraeus-datastore.svc:443/validate-piraeus-io-v1-linstorcluster?timeout=10s": proxy error from 127.0.0.1:6443 while dialing 10.42.3.14:9443, code 502: 502 Bad Gateway

[WanzenBug]

This is an issue with your Kubernetes API Server. The API Server is trying to reach the webhook, but for some reason it can't. Perhaps look at the API Server logs to find out more.