GitHubSecretEngineConfig referencing vaultSecret fails with panic error

[rafaeltuelho]

Here is my CRD:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: GitHubSecretEngineConfig
metadata:
  name: github2
  namespace: vault-admin
spec:
  authentication: 
    path: kubernetes
    role: policy-admin
  sSHKeyReference:
    vaultSecret:
      path: "kv/data/secrets/janusidp/github-plugin2"
  path: github
  applicationID: 111111

I see this error on vault-admin operator pod:

2023-11-06T16:22:19Z	INFO	Observed a panic in reconciler: interface conversion: interface {} is nil, not string	{"controller": "githubsecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "GitHubSecretEngineConfig", "GitHubSecretEngineConfig": {"name":"github2","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "github2", "reconcileID": "a4b2534d-2c5c-4441-8bfe-e688c0619278"}
panic: interface conversion: interface {} is nil, not string [recovered]
	panic: interface conversion: interface {} is nil, not string

goroutine 946 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:115 +0x1fa
panic({0x18a3560, 0xc003778de0})
	/opt/hostedtoolcache/go/1.19.13/x64/src/runtime/panic.go:884 +0x212
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*GitHubSecretEngineConfig).setInternalCredentials(0xc00067ec40, {0x1d80ca8, 0xc0059c52f0})
	/home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/githubsecretengineconfig_types.go:156 +0x5dd
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*GitHubSecretEngineConfig).PrepareInternalValues(0x1d80ca8?, {0x1d80ca8?, 0xc0059c52f0?}, {0x0?, 0x0?})
	/home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/githubsecretengineconfig_types.go:116 +0x25
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic(0xc003729bb0, {0x1d80ca8, 0xc0059c52f0}, {0x1d94100?, 0xc00067ec40?})
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:93 +0xb6
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile(0xc003729bb0, {0x1d80ca8, 0xc0059c52f0}, {0x1d94100, 0xc00067ec40})
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65 +0x239
github.com/redhat-cop/vault-config-operator/controllers.(*GitHubSecretEngineConfigReconciler).Reconcile(0xc000200380, {0x1d80ca8, 0xc0022a1bf0}, {{{0xc0009689a0?, 0x10?}, {0xc000968996?, 0x40f6e7?}}})
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/githubsecretengineconfig_controller.go:85 +0x3f8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d80ca8?, {0x1d80ca8?, 0xc0022a1bf0?}, {{{0xc0009689a0?, 0x1806740?}, {0xc000968996?, 0x10?}}})

And here is the secret path on vault:

 vault kv get -mount=kv /secrets/janusidp/github-plugin2
============= Secret Path =============
kv/data/secrets/janusidp/github-plugin2

======= Metadata =======
Key                Value
---                -----
created_time       2023-11-06T16:57:17.834324875Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
key    -----DATA-----

It seems the errors happens at this line:

vault-config-operator/api/v1alpha1/githubsecretengineconfig_types.go

Line 156 in a0f0390

r.Spec.retrievedSSHKey = secret.Data["key"].(string)

[raffaelespazzoli]

it looks like the path you are passing is wrong (the data segment seems to be at the wrong place) and at the same time, the operator does not handle that situation gracefully.
Thanks for reporting this.

[rafaeltuelho]

So, I enabled my KV using v2, so that's why it adds /data to the mount point kv. Does the Operator expect kv v1 (without versioning)?

[rafaeltuelho]

I ran into another situation where the Operator crashes with panic error.
This happens when it tries to read a secret that does not exist in the vault yet.

2023-11-30T21:19:41Z	DEBUG	reconcile	{"controller": "secretenginemount", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "SecretEngineMount", "SecretEngineMount": {"name":"github","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "github", "reconcileID": "42c56db7-0989-4a87-ac90-60da4b9ec9a8", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "SecretEngineMount", "namespace": "vault-admin", "name": "github"}}
2023-11-30T21:19:41Z	ERROR	unable to retrieve vault secret	{"controller": "quaysecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "QuaySecretEngineConfig", "QuaySecretEngineConfig": {"name":"quay","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "quay", "reconcileID": "1bfca365-fab9-4923-aa8e-7faca0920e8f", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "QuaySecretEngineConfig", "namespace": "vault-admin", "name": "quay"}, "error": "secret not found"}
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*QuaySecretEngineConfig).setInternalCredentials
	/home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/quaysecretengineconfig_types.go:138
github.com/redhat-cop/vault-config-operator/api/v1alpha1.(*QuaySecretEngineConfig).PrepareInternalValues
	/home/runner/work/vault-config-operator/vault-config-operator/api/v1alpha1/quaysecretengineconfig_types.go:81
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:93
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65
github.com/redhat-cop/vault-config-operator/controllers.(*QuaySecretEngineConfigReconciler).Reconcile
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/quaysecretengineconfig_controller.go:76
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
2023-11-30T21:19:41Z	ERROR	unable to prepare internal values	{"controller": "quaysecretengineconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "QuaySecretEngineConfig", "QuaySecretEngineConfig": {"name":"quay","namespace":"vault-admin"}, "namespace": "vault-admin", "name": "quay", "reconcileID": "1bfca365-fab9-4923-aa8e-7faca0920e8f", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "QuaySecretEngineConfig", "namespace": "vault-admin", "name": "quay"}, "error": "secret not found"}
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).manageReconcileLogic
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:95
github.com/redhat-cop/vault-config-operator/controllers/vaultresourcecontroller.(*VaultResource).Reconcile
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/vaultresourcecontroller/vaultresourcereconciler.go:65
github.com/redhat-cop/vault-config-operator/controllers.(*QuaySecretEngineConfigReconciler).Reconcile
	/home/runner/work/vault-config-operator/vault-config-operator/controllers/quaysecretengineconfig_controller.go:76
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226

I believe in such cases, the Controller should handle the error, log it, and move on.

[raffaelespazzoli]

the first one is a bug, we should in fact check for the key key to exist and not panic.
The second does not seem to be an error. the operator does what you said: reports an error and move on.
In both cases the errors occur because the secrets are not populated (yet).

[rafaeltuelho]

Yeah, in the second scenario I ran into the Operator POD crashed and stayed unrecovered. Yes, the secret was not populated yet in Vault. So, I think the operator should not crash and just ignore and keep trying to check the secret in vault.