Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to upgrade kubernetes version of AKS imported cluster with Network plugin type Azure CNI #294

Closed
2 tasks done
cpinjani opened this issue Oct 17, 2023 · 3 comments
Closed
2 tasks done

Unable to upgrade kubernetes version of AKS imported cluster with Network plugin type Azure CNI #294

cpinjani opened this issue Oct 17, 2023 · 3 comments
Assignees
@mjura @valaparthvi
Labels
area/network kind/bug Something isn't working
Milestone
v2.8.7

Comments

@cpinjani
Copy link
Contributor

cpinjani commented Oct 17, 2023 •
edited by mjura
Loading

Rancher version:

2.8.0-rc1
aks-operator:v1.2.0-rc4

Describe the bug
Unable to upgrade kubernetes version for AKS imported cluster with Network plugin type Azure CNI
It is working for Network plugin type Kubenet

Submited PR's

Steps

  1. Create AKS cluster on Azure portal with k8s version one less than supported and Network plugin type Azure CNI, Network policy - Any
  2. Import above AKS cluster in Rancher and wait for it to be Active
  3. Edit cluster and upgrade version to highest supported

image

Logs

time="2023-10-17T09:19:59Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:00Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:09Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:20:10Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:11Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:19Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:20:20Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:21Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:29Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:20:30Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:31Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:40Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:20:41Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:42Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:50Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:20:51Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:20:52Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:00Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:01Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:02Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:11Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:12Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:13Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:23Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:24Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:25Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:33Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:34Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:36Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:45Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:46Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:47Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:56Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:21:56Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:21:57Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:06Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:22:07Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:09Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:17Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:22:18Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:19Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:28Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:22:38Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:40Z" level=info msg="Updating kubernetes version for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:48Z" level=error msg="error syncing 'cattle-global-data/c-cgszl': handler aks-controller: failed to update cluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"PropertyChangeNotAllowed\" Message=\"Changing property 'networkProfile.networkPlugin' is not allowed.\" Target=\"networkProfile.networkPlugin\", requeuing"
time="2023-10-17T09:22:48Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:49Z" level=info msg="Cluster [c-cgszl] finished updating"
time="2023-10-17T09:22:51Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:52Z" level=info msg="Configuration for cluster [cpinjani-imp1] was verified"
time="2023-10-17T09:22:52Z" level=error msg="Error recording akscc [] failure message: Operation cannot be fulfilled on aksclusterconfigs.aks.cattle.io \"c-cgszl\": the object has been modified; please apply your changes to the latest version and try again"
time="2023-10-17T09:22:53Z" level=info msg="Checking configuration for cluster [cpinjani-imp1]"
time="2023-10-17T09:22:54Z" level=info msg="Configuration for cluster [cpinjani-imp1] was verified"
@cpinjani cpinjani added kind/bug Something isn't working team/highlander labels Oct 17, 2023
@cpinjani cpinjani added this to the v2.8.0 milestone Oct 19, 2023
@kkaempf
Copy link

kkaempf commented Oct 19, 2023

Let's see what the root cause is and if there's a simple workaround.

Otherwise I'd rather move this to 2024-Q1

@mjura mjura moved this from Backlog to In Progress (5 max) in CAPI & Hosted Kubernetes providers (EKS/AKS/GKE) Oct 24, 2023
@mjura mjura self-assigned this Oct 24, 2023
@kkaempf kkaempf modified the milestones: v2.8.0, 2024-Q1-v2.8x Oct 25, 2023
mjura added a commit to mjura/aks-operator that referenced this issue Oct 31, 2023
@mjura
Fix for updating imported clusters
4de0110 
Issue: rancher#294
Signed-off-by: Michal Jura <[email protected]>
mjura added a commit to mjura/aks-operator that referenced this issue Oct 31, 2023
@mjura
Fix for updating imported clusters
f0d1c23 
Issue: rancher#294
(cherry picked from commit 1bbce8e781ae8adf387d3c22e5c33d457f4a95a4)
This was referenced Oct 31, 2023
Merged
Merged
@mjura mjura moved this from In Progress (5 max) to PR to be reviewed in CAPI & Hosted Kubernetes providers (EKS/AKS/GKE) Oct 31, 2023
@mjura mjura moved this from PR to be reviewed to To Test in CAPI & Hosted Kubernetes providers (EKS/AKS/GKE) Nov 3, 2023
@valaparthvi valaparthvi self-assigned this Dec 13, 2023
@valaparthvi
Copy link
Contributor

Tested this on aks-operator(ttl.sh/aks-operator-nightly-20231213:1d) and Rancher(v2.8-ea5516dac7b1d191116c33232fc307a47cd48d84-head).
Upgrading to the latest version was successful in the end, but there were some errors in the log.

Logs 

$ k logs -n cattle-system aks-config-operator-6c4899b449-2dtdz
time="2023-12-14T12:31:23Z" level=info msg="Starting aks.cattle.io/v1, Kind=AKSClusterConfig controller"
time="2023-12-14T12:31:23Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-12-14T12:44:51Z" level=info msg="Importing config for cluster [pvala-aks-cni]"
time="2023-12-14T12:44:58Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:45:00Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"
time="2023-12-14T12:46:38Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:46:40Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:16Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:18Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:24Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:47:26Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:28Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:34Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:47:37Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:39Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:45Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:47:47Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:49Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:55Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:47:57Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:47:59Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:05Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:48:07Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:09Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:15Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:48:16Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:18Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:34Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:48:36Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:38Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:45Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:48:47Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:49Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:48:55Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:48:57Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:03Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:09Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:49:11Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:13Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:19Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:49:21Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:23Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:28Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:49:30Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:32Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:38Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:49:41Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:43Z" level=info msg="Updating max surge in node pool [userpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:49:48Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:50:04Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:50:06Z" level=info msg="Cluster [c-hkjth] finished updating"
time="2023-12-14T12:50:08Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:50:10Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"
time="2023-12-14T12:50:20Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:50:22Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"
time="2023-12-14T12:58:12Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:14Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:33Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:34Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:39Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:58:41Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:43Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:47Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:58:49Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:51Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:55Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:58:57Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:58:58Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:03Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:04Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:07Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:11Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:13Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:15Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:20Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:22Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:23Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:28Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:30Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:32Z" level=info msg="Updating max surge in node pool [userpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:40Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:42Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:44Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:49Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:50Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:52Z" level=info msg="Updating max surge in node pool [userpool] for cluster [pvala-aks-cni]"
time="2023-12-14T12:59:57Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T12:59:59Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:01Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:06Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T13:00:07Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:09Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:17Z" level=error msg="error syncing 'cattle-global-data/c-hkjth': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2023-12-14T13:00:19Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:22Z" level=info msg="Cluster [c-hkjth] finished updating"
time="2023-12-14T13:00:24Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:26Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"
time="2023-12-14T13:00:26Z" level=error msg="Error recording akscc [] failure message: Operation cannot be fulfilled on aksclusterconfigs.aks.cattle.io \"c-hkjth\": the object has been modified; please apply your changes to the latest version and try again"
time="2023-12-14T13:00:28Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:32Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"
time="2023-12-14T13:00:34Z" level=info msg="Checking configuration for cluster [pvala-aks-cni]"
time="2023-12-14T13:00:36Z" level=info msg="Configuration for cluster [pvala-aks-cni] was verified"

@valaparthvi
Copy link
Contributor

valaparthvi commented Jan 4, 2024 •
edited
Loading

Tested this again on the nightly aks-operator. I do not see the error that Chandan mentioned in the description, and I was able to successfully upgrade AKS k8s version.

aks-operator logs 

➜  hosted-providers-e2e git:(add-zone-to-input) k logs -n cattle-system aks-config-operator-84fdbfc6-ps7vv --follow
time="2024-01-04T07:36:07Z" level=info msg="Starting aks.cattle.io/v1, Kind=AKSClusterConfig controller"
time="2024-01-04T07:36:07Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2024-01-04T07:37:50Z" level=info msg="Importing config for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:37:54Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:37:56Z" level=info msg="Configuration for cluster [pvala-aks-azure-cni-k8s-upgrade] was verified"
time="2024-01-04T07:43:19Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:20Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:20Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:20Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:21Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:23Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:23Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:23Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:24Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:25Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:25Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:25Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:26Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:28Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:28Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:28Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:29Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:30Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:30Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:30Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:32Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:33Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:33Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:33Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:34Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:35Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:35Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:35Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:36Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:38Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:38Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:38Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:39Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:40Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:41Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:41Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:42Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:43Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:43Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:43Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:44Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:45Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:45Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:45Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:43:47Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:48Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:43:48Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:43:48Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:44:00Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:01Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:01Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:44:01Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:44:23Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:24Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:24Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:44:24Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:44:55Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:56Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:44:56Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:44:56Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:45:27Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:45:28Z" level=info msg="Updating kubernetes version for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:45:28Z" level=info msg="Create Azure Log Analytics Workspace \"DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS\" on Resource Group \"DefaultResourceGroup-EUS\""
time="2024-01-04T07:45:28Z" level=error msg="error syncing 'cattle-global-data/c-jgz8r': handler aks-controller: failed to update cluster: operationalinsights.WorkspacesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code=\"AuthorizationFailed\" Message=\"The client '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' with object id '080af9a2-7d62-4e17-a72a-ea80c6ac38fd' does not have authorization to perform action 'Microsoft.OperationalInsights/workspaces/write' over scope '/subscriptions/80de5134-ca65-4731-be21-13fb1f0910a2/resourcegroups/DefaultResourceGroup-EUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-80de5134-ca65-4731-be21-13fb1f0910a2-EUS' or the scope is invalid. If access was recently granted, please refresh your credentials.\", requeuing"
time="2024-01-04T07:45:39Z" level=info msg="Cluster [pvala-aks-azure-cni-k8s-upgrade] is imported, will not delete AKS cluster"
time="2024-01-04T07:47:07Z" level=info msg="Importing config for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:47:10Z" level=info msg="Checking configuration for cluster [pvala-aks-azure-cni-k8s-upgrade]"
time="2024-01-04T07:47:12Z" level=info msg="Configuration for cluster [pvala-aks-azure-cni-k8s-upgrade] was verified"
time="2024-01-04T07:58:10Z" level=info msg="Cluster [pvala-aks-azure-cni-k8s-upgrade] is imported, will not delete AKS cluster"
time="2024-01-04T09:02:56Z" level=info msg="Importing config for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:03:02Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:03:03Z" level=info msg="Configuration for cluster [pvala-k8s-azure-cni] was verified"
time="2024-01-04T09:07:06Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:08Z" level=info msg="Updating kubernetes version for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:26Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:28Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:33Z" level=error msg="error syncing 'cattle-global-data/c-xrw82': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2024-01-04T09:07:35Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:37Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:42Z" level=error msg="error syncing 'cattle-global-data/c-xrw82': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2024-01-04T09:07:43Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:45Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:51Z" level=error msg="error syncing 'cattle-global-data/c-xrw82': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2024-01-04T09:07:52Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:07:54Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:00Z" level=error msg="error syncing 'cattle-global-data/c-xrw82': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2024-01-04T09:08:02Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:03Z" level=info msg="Updating max surge in node pool [agentpool] for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:08Z" level=error msg="error syncing 'cattle-global-data/c-xrw82': handler aks-controller: failed to update cluster: containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>, requeuing"
time="2024-01-04T09:08:10Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:12Z" level=info msg="Cluster [c-xrw82] finished updating"
time="2024-01-04T09:08:14Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:16Z" level=info msg="Configuration for cluster [pvala-k8s-azure-cni] was verified"
time="2024-01-04T09:08:16Z" level=error msg="Error recording akscc [] failure message: Operation cannot be fulfilled on aksclusterconfigs.aks.cattle.io \"c-xrw82\": the object has been modified; please apply your changes to the latest version and try again"
time="2024-01-04T09:08:18Z" level=info msg="Checking configuration for cluster [pvala-k8s-azure-cni]"
time="2024-01-04T09:08:20Z" level=info msg="Configuration for cluster [pvala-k8s-azure-cni] was verified"

@cpinjani I am closing this issue, please reopen it if you feel the need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mjura mjura

@valaparthvi valaparthvi

Labels
area/network kind/bug Something isn't working
Projects
CAPI & Hosted Kubernetes providers (EKS/AKS/GKE)
Archived in project
Milestone
v2.8.7
Development

No branches or pull requests
4 participants
@kkaempf @mjura @valaparthvi @cpinjani