F5-cis needs a restart to use updated kubeconfig secret

[achaudh25]

Setup Details

CIS Version : 2.19
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP v17.1.1.3
AS3 Version: 3.52
Agent Mode: AS3
Orchestration: K8S
Orchestration Version:
Pool Mode: Nodeport
Additional Setup details: Rancher/Calico network

Description

When kubeconfig secret changes f5-cis still keep using the cached old secret. To update the secret f5-cis needs a restart.

Steps To Reproduce

1. Update the secret with dummy secret on both primary and secondory cluster (multicluster f5-cis)
2. F5-cis no error is connection between both cluster.
3. Update the transport server changes pushed and all pool member green from both cluster.
4. Restart pod on primary cluster and start seeing error:
   Failed to watch CustomPolicy in secondary cluster User system:unauthenticated cannot get resources
5. All pool member on transport server secondory cluster down.
6. Repeat the steps with original kubeconfig secret and had to restart the f5-cis pod to recover from above error.

Expected Result

Change in kubeconfig secret detected by f5-cis and no restart needed.

[trinaths]

Created [CONTCNTR-5196] for internal tracking.

[lavanya-f5]

@achaudh25 Hi, need some information to reproduce the issue. can you confirm, if the kubeconfig is expired for the cluster, is the secret updated with new kubeconfig ? or secret with old kubeconfig is deleted and created with new kubeconfig?

[achaudh25]

1. Update the secret with wrong/expired kubeconfig
   ex: cluster1_kubeconfig and cluster2_ kubeconfig : === updated them on both the clusters with wrong/expired kubeconfig
2. No impact seen in the pod logs untill restarted the pod
3. Update the same secrets with kubeconfig with was originally there and expiring in next 40 days.
4. Keep on seeing same error : Failed to watch CustomPolicy in secondary cluster User system:unauthenticated cannot get resources
5. Restarted the pod and no issues it was able to connect with secondary clsuter.

[lavanya-f5]

Hi avani, thanks for the info.

1. Update the secret with wrong/expired kubeconfig
   ex: cluster1_kubeconfig and cluster2_ kubeconfig : === updated them on both the clusters with wrong/expired kubeconfig
   ----- Able to reproduce issue with kubeconfig secret updates for external clusters(including secondary cluster), this is resolved

dev build: lavanyasirigudi/k8s-bigip-ctlr:kubeconfig_update_fix

For primary clusters(i.e local cluster) where cis is running, kubeconfig is read from inClusterConfig provided by Kubernetes. Secret update will have no impact on local cluster.

Please share your feedback with the build.

[achaudh25]

We tested the new build in our environment and it worked for us. Thanks for help

[achaudh25]

Just a follow up question when this will be released.