Custom resource readiness with self-signed TLS on Openshift

[csotiistvan]

Hi,

We are running into a weird issue on Openshift with TLS enabled when using certificates with a self-signed CA. This is an air-gapped environment and we are forced to use a self-signed CA to issue our TLS certificates. What we are experiencing is that Queue resources have their readiness ping-ponging between Ready and Failed at a 1:2 ratio every couple of seconds. When the queue declaration fails, the reason is always tls: failed to verify certificate: x509: certificate signed by unknown authority .

Our CA is both included in the Openshift config-trusted-cabundle and mounted as described in the TLS guide. Both the operator pod and the server pod have our CA mounted to the specified location. Installation was done as per these instructions, our caBundle inserted into all of the webhooks as explained there. Also tried the Operatorhub version, but we could not insert our own CA in that case, same problem as here.

Seems to me like there are either multiple webhooks checking the Queues and some are http and others https, or some have our CA and others lack it. Verified the CA and TLS certs are fine, I can curl the https api endpoint when our CA is in a pod's CA bundle. We have the correct alt names for the webhook service in the certificate.

Any help with this would be much appreciated, it's like banging my head into a wall in the last couple days.

Cheers all!

[DanielePalaia]

Hi @csotiistvan,

are you still experimenting the issue?

I made a fast test on my openshift environment and I wasn't able to reproduce it.

I generated the certificates using this script: https://github.com/rabbitmq/tls-gen

The hostname should be webhook-service.rabbitmq-system.svc so I make with

make CN=webhook-service.rabbitmq-system.svc

This will create the certificates needed in the result folder.

I then created the secret:

kubectl create secret generic webhook-server-cert -n rabbitmq-system --from-file=ca.crt=./ca_certificate.pem --from-file=tls.crt=./server_webhook-service.rabbitmq-system.svc_certificate.pem --from-file=tls.key=server_webhook-service.rabbitmq-system.svc_key.pem

and then editing the messaging operator manifest putting the base64 of the ca_cert in the caBundle field as in the doc (I tried just to configure queues and creating/deleting/modifying a queue object looks good).

Indeed when I try to add a non configured object (where caBundle is not present) I get:

Error from server (InternalError): error when creating "user.yml": Internal error occurred: failed calling webhook "vuser.kb.io": failed to call webhook: Post "https://webhook-service.rabbitmq-system.svc:443/validate-rabbitmq-com-v1beta1-user?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

which is normal as I didn't configure caBundle for that object.

It seems like you are having the issue intermittently?

Regarding the OLM distribution instead yes it is something to investigated. Apprently OLM automatically create and mount them:

https://olm.operatorframework.io/docs/advanced-tasks/adding-admission-and-conversion-webhooks/

There is no guideline on how to do this manually on a csv. It something that we need still to research.