Controller unreachable after cert-manager certificates expire

[kmorning]

Environment details

• Kubernetes v1.29.4
• pireaeus-operator 2.5.1
• cert-manager v1.14.5

Description
I've setup both the internal and api TLS using certmanager as described in how-to docs. It seems that the certificates are valid for 3 month upon issue. Certmanager renews the certificates automatically 1 month before expiry, but the deployed linstor pods still use the old certificates. When the original certificates expire, cluster communications are broken. I had to manually delete the controller, satellite and csi pods so that they spin up again using the renewed certs.

Desired behavior
Controller, satellite and csi pods should reload certs or restart after renewal.

[WanzenBug]

I guess non of our components implement watching and reloading of the certificates.

[kmorning]

As an interim solution, I've deployed Reloader to my cluster, that watches for changes in dependent ConfigMap or Secret resources for deployments, daemonsets, or statefulsets, and performs a rolling update on pods when changes are seen.

All that needs to be done is to patch the relevant linstor resources with the annotation required for the reloader controller to watch them. My LinstorCluster and LinstorSatelliteConfiguration resources now look like this:

apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  internalTLS:
    certManager:
      name: linstor-internal-ca
      kind: Issuer
  apiTLS:
    certManager:
      name: linstor-api-ca
      kind: Issuer
  patches:
    - target:
        kind: Deployment
        name: linstor-controller
      patch: |-
        - op: add
          path: /metadata/annotations/reloader.stakater.com~1auto
          value: "true"
    - target:
        kind: Deployment
        name: linstor-csi-controller
      patch: |-
        - op: add
          path: /metadata/annotations/reloader.stakater.com~1auto
          value: "true"
    - target:
        kind: DaemonSet
        name: linstor-csi-node
      patch: |-
        - op: add
          path: /metadata/annotations/reloader.stakater.com~1auto
          value: "true"

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: satellite-reloader
spec:
  patches:
    - target:
        kind: DaemonSet
        name: linstor-satellite
      patch: |-
        - op: add
          path: /metadata/annotations/reloader.stakater.com~1auto
          value: "true"

It would still probably be better for piraeus-operator to handle the watch of certs and not have to deploy an additional resource to the cluster. Reloader is pretty handy for other applications outside of this particular use case as well though.

[xy2]

I would like this to be implemented in the Piraeus operator. Each Satellite pod runs in its own daemonset. As a result, all satellites can be restarted simultaneously by Stakater Reloader. If something goes wrong during restart, we get all satellites down instead of one. Piraeus operator could handle that situation better since it knows more about Satellites.