OCPBUGS-48250: MCO CO degrades are stuck on until master pool updates complete

[djoshy]

- What I did
Added a clear CO degrade function, which is called after a successful invocation of an operator sync function.

- How to verify it
On a build without this fix:

1. Degrade the operator. I did this is by scaling down the CVO and editing the releaseVersion field in the machine-config-operator-images configmap to a bad value. This will cause syncRenderConfig to fail and degrade the operator(visible in the CO object and operator logs).
2. Now, deploy an MC update to the master pool. This will cause the operator to be stuck in syncRequiredMachineConfigPool sync function, where it'll wait until the master pool completes the update.
3. While the master pool is still updating, restore the releaseVersion back to the original value. You should see the operator log clear up shortly, but the CO will continue to be degraded. Once the master pool is done updating, the CO degrade will clear up.

On a build with this fix:
Repeat steps 1 to 4 above. This time, you should notice that the CO degrade will clear up shortly after restoring releaseVersion, without having to wait for the master pool to complete the update.

Note: The update needs to be applied to a master pool because the syncRequiredMachineConfigPools function will only "trap" the operator for master pool updates.

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-48250, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did
Added a clear CO degrade function, which is called after a successful invocation of an operator sync function.

- How to verify it
On a build without this fix:

1. Degrade the operator. I did this is by scaling down the CVO and editing the releaseVersion field in the machine-config-operator-images configmap to a bad value. This will cause syncRenderConfig to fail and degrade the operator(visible in the CO object and operator logs).
2. Now, deploy an MC update to the master pool. This will cause the operator to be stuck in syncRequiredMachineConfigPool sync function, where it'll wait until the master pool completes the update.
3. While the master pool is still updating, restore the releaseVersion back to the original value. You should see the operator log clear up shortly, but the CO will continue to be degraded. Once the master pool is done updating, the CO degrade will clear up.

On a build with this fix:
Repeat steps 1 to 4 above. This time, you should notice that the CO degrade will clear up shortly after restoring releaseVersion, without having to wait for the master pool to complete the update.

Note: The update needs to be applied to a master pool because the syncRequiredMachineConfigPools function will only "trap" the operator for master pool updates.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/retest-required

[djoshy]

/test security

[sergiordlr]

Verified using IPI on AWS

1. Scale down CVO
   $ oc scale deployment cluster-version-operator --replicas 0 -n openshift-cluster-version

2. Remove the configuration of the machine-config-operator-images configmap to force a degraded status in the machine-config clusteroperator

# Don't forget the get the original configuration, we will restore it later
$ oc get cm machine-config-operator-images -oyaml
$ oc set data cm machine-config-operator-images 'images.json='

3. Wait until the machine-config CO is degraded

$ oc get co machine-config
NAME             VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
machine-config   4.19.0-0.test-2025-01-20-085829-ci-ln-x2x5hkk-latest   True        False         True       120m    Failed to resync 4.19.0-0.test-2025-01-20-085829-ci-ln-x2x5hkk-latest because: could not parse images.json bytes: unexpected end of JSON input

4. Apply a mc to the master pool

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: test-mc-master
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,dGVzdA==
        filesystem: root
        mode: 420
        path: /etc/test-file-0.test

5. Wait for the master pool to start updating

$ oc get mcp,nodes; oc get co machine-config
NAME                                                         CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
machineconfigpool.machineconfiguration.openshift.io/master   rendered-master-55d74396e13f62dc07bc3b38e3ed2aa3   False     True       False      3              0                   0                     0                      128m
machineconfigpool.machineconfiguration.openshift.io/worker   rendered-worker-639975f0322cd83401ab1d9a80e6ae46   True      False      False      3              3                   3                     0                      128m

NAME                                             STATUS                     ROLES                  AGE    VERSION
node/ip-10-0-11-207.us-east-2.compute.internal   Ready                      worker                 125m   v1.31.3
node/ip-10-0-21-105.us-east-2.compute.internal   Ready,SchedulingDisabled   control-plane,master   130m   v1.31.3
node/ip-10-0-34-100.us-east-2.compute.internal   Ready                      control-plane,master   130m   v1.31.3
node/ip-10-0-51-58.us-east-2.compute.internal    Ready                      worker                 125m   v1.31.3
node/ip-10-0-64-184.us-east-2.compute.internal   Ready                      worker                 125m   v1.31.3
node/ip-10-0-84-199.us-east-2.compute.internal   Ready                      control-plane,master   130m   v1.31.3

5. Fix the machine-config-operator-images configmap using the original configuration that we got in step 2
6. After a few seconds the machine-config CO should stop being degraded

$ oc get co machine-config
NAME             VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
machine-config   4.19.0-0.test-2025-01-20-085829-ci-ln-x2x5hkk-latest   True        False         False      127m   

/label qe-approved

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-48250, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

- What I did
Added a clear CO degrade function, which is called after a successful invocation of an operator sync function.

- How to verify it
On a build without this fix:

1. Degrade the operator. I did this is by scaling down the CVO and editing the releaseVersion field in the machine-config-operator-images configmap to a bad value. This will cause syncRenderConfig to fail and degrade the operator(visible in the CO object and operator logs).
2. Now, deploy an MC update to the master pool. This will cause the operator to be stuck in syncRequiredMachineConfigPool sync function, where it'll wait until the master pool completes the update.
3. While the master pool is still updating, restore the releaseVersion back to the original value. You should see the operator log clear up shortly, but the CO will continue to be degraded. Once the master pool is done updating, the CO degrade will clear up.

On a build with this fix:
Repeat steps 1 to 4 above. This time, you should notice that the CO degrade will clear up shortly after restoring releaseVersion, without having to wait for the master pool to complete the update.

Note: The update needs to be applied to a master pool because the syncRequiredMachineConfigPools function will only "trap" the operator for master pool updates.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, isabella-janssen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[djoshy]

/label acknowledge-critical-fixes-only

should be a safe change, this should just improve overall speed with which CO degrades clear

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD b2e4422 and 2 for PR HEAD 07e371a in total

[djoshy]

/test unit

[openshift-ci]

@djoshy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vsphere-ovn-upi	07e371a	link	false	/test e2e-vsphere-ovn-upi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[djoshy]

/retest-required

[yuqi-zhang]

/override ci/prow/e2e-hypershift

Isn't affected (and passed in the past)

[openshift-ci]

@yuqi-zhang: Overrode contexts on behalf of yuqi-zhang: ci/prow/e2e-hypershift

In response to this:

/override ci/prow/e2e-hypershift

Isn't affected (and passed in the past)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@djoshy: Jira Issue OCPBUGS-48250: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#4791

Jira Issue OCPBUGS-48250 has been moved to the MODIFIED state.

In response to this:

- What I did
Added a clear CO degrade function, which is called after a successful invocation of an operator sync function.

- How to verify it
On a build without this fix:

1. Degrade the operator. I did this is by scaling down the CVO and editing the releaseVersion field in the machine-config-operator-images configmap to a bad value. This will cause syncRenderConfig to fail and degrade the operator(visible in the CO object and operator logs).
2. Now, deploy an MC update to the master pool. This will cause the operator to be stuck in syncRequiredMachineConfigPool sync function, where it'll wait until the master pool completes the update.
3. While the master pool is still updating, restore the releaseVersion back to the original value. You should see the operator log clear up shortly, but the CO will continue to be degraded. Once the master pool is done updating, the CO degrade will clear up.

On a build with this fix:
Repeat steps 1 to 4 above. This time, you should notice that the CO degrade will clear up shortly after restoring releaseVersion, without having to wait for the master pool to complete the update.

Note: The update needs to be applied to a master pool because the syncRequiredMachineConfigPools function will only "trap" the operator for master pool updates.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/cherry-pick release-4.18 release-4.17

[openshift-cherrypick-robot]

@djoshy: new pull request created: #4818

In response to this:

/cherry-pick release-4.18 release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.19.0-202501291707.p0.g31c9683.assembly.stream.el9.
All builds following this will include this PR.