OCPBUGS-48340: skip OperatorHubSourceError metric checking when disableAllDefaultSources is true #29435

jianzhangbjz · 2025-01-14T07:39:40Z

When disableAllDefaultSources is true, all default catalogSource are disabled, so there is no metric.

    [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "OperatorHubSourceError",
          "alertstate": "firing",
          "container": "catalog-operator",
          "endpoint": "https-metrics",
          "exported_namespace": "openshift-marketplace",
          "instance": "[fd01:0:0:1::1a]:8443",
          "job": "catalog-operator-metrics",
          "name": "community-operators",
          "namespace": "openshift-operator-lifecycle-manager",
          "pod": "catalog-operator-6c446dcbbb-sxvjz",
          "prometheus": "openshift-monitoring/k8s",
          "service": "catalog-operator-metrics",
          "severity": "warning"
        },
        "value": [
          1736751753.045,
          "1"
        ]
      }
    ]

openshift-ci-robot · 2025-01-14T07:39:46Z

@jianzhangbjz: This pull request references Jira Issue OCPBUGS-48340, which is invalid:

expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

When disableAllDefaultSources is true, all default catalogSource are disabled, so there is no metric.

   [
     {
       "metric": {
         "__name__": "ALERTS",
         "alertname": "OperatorHubSourceError",
         "alertstate": "firing",
         "container": "catalog-operator",
         "endpoint": "https-metrics",
         "exported_namespace": "openshift-marketplace",
         "instance": "[fd01:0:0:1::1a]:8443",
         "job": "catalog-operator-metrics",
         "name": "community-operators",
         "namespace": "openshift-operator-lifecycle-manager",
         "pod": "catalog-operator-6c446dcbbb-sxvjz",
         "prometheus": "openshift-monitoring/k8s",
         "service": "catalog-operator-metrics",
         "severity": "warning"
       },
       "value": [
         1736751753.045,
         "1"
       ]
     }
   ]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jianzhangbjz · 2025-01-14T07:41:15Z

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

simonpasquier · 2025-01-14T08:31:50Z

this isn't the correct approach, you should rather add the alert name to AllowedAlertNames in pkg/monitortestlibrary/allowedalerts/types.go (with a link to the bug) to unblock the situation. Then work to fix the alert expression so it doesn't fire in the first place when OperatorHub is disabled.

jianzhangbjz · 2025-01-14T09:20:14Z

Hi @simonpasquier , thanks for the help! Could you help review it again? Thanks!

jianzhangbjz · 2025-01-14T09:20:27Z

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

simonpasquier · 2025-01-14T09:23:09Z

pkg/monitortestlibrary/allowedalerts/types.go

@@ -47,4 +47,5 @@ var AllowedAlertNames = []string{
 	"CDIDefaultStorageClassDegraded", // Installing openshift virt with RWX storage fire an alarm, that is not relevant for most of the tests.
 	"VirtHandlerRESTErrorsHigh",      // https://issues.redhat.com/browse/CNV-50418
 	"VirtControllerRESTErrorsHigh",   // https://issues.redhat.com/browse/CNV-50418
+	"OperatorHubSourceError",         // https://issues.redhat.com/browse/OCPBUGS-48340, should skip this metric checking when disableAllDefaultSources is true


Suggested change

"OperatorHubSourceError", // https://issues.redhat.com/browse/OCPBUGS-48340, should skip this metric checking when disableAllDefaultSources is true

"OperatorHubSourceError", // https://issues.redhat.com/browse/OCPBUGS-48340

simonpasquier · 2025-01-14T09:23:35Z

test/extended/prometheus/prometheus.go

+			// https://issues.redhat.com/browse/OCPBUGS-48340
+			if SkipOperatorHubMetricsCheck(oc) {
+				allowedAlertNames = removeElement(allowedAlertNames, "OperatorHubSourceError")
+			}


no need for this

simonpasquier · 2025-01-14T09:23:44Z

test/extended/prometheus/prometheus.go

+
+func SkipOperatorHubMetricsCheck(oc *exutil.CLI) bool {
+	stdout, stderr, err := oc.AsAdmin().Run("get").Args("operatorhub", "cluster", "-o=jsonpath={.spec.disableAllDefaultSources}").Outputs()
+	if err != nil {
+		fmt.Printf("command failed: %v\nstderr: %s\nstdout:%s", err, stderr, stdout)
+	}
+	return stdout == "true"
+}
+
+func removeElement(slice []string, element string) []string {
+	var result []string
+	for _, v := range slice {
+		if v != element {
+			result = append(result, v)
+		}
+	}
+	return result
+}


…rces is true

jianzhangbjz · 2025-01-14T09:34:09Z

Hi @simonpasquier , could you help review it again? Thanks! The logic is that add OperatorHubSourceError to allowedAlertNames when SkipOperatorHubMetricsCheck is true.

joelanford · 2025-01-14T20:35:32Z

@jianzhangbjz Can you explain the rationale here more? This alert is only firing 7% of the time right now. Where are the default sources being disabled in CI clusters?

jianzhangbjz · 2025-01-15T01:44:26Z

Hi @joelanford , I believe they disable it in the baremetalds-devscripts-setup step, see the https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-upgrade-ovn-ipv6/1877545344619778048/artifacts/e2e-metal-ipi-upgrade-ovn-ipv6/baremetalds-devscripts-setup/artifacts/root/dev-scripts/logs/06_create_cluster-2025-01-09-205310.log

2025-01-09 21:45:15 +(utils.sh:754): add_local_certificate_as_trusted(): oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge
2025-01-09 21:45:15 image.config.openshift.io/cluster patched
2025-01-09 21:45:15 +(./06_create_cluster.sh:81): [[ -n true ]]
2025-01-09 21:45:15 +(./06_create_cluster.sh:81): [[ true != \f\a\l\s\e ]]
2025-01-09 21:45:15 +(./06_create_cluster.sh:82): oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
2025-01-09 21:45:16 operatorhub.config.openshift.io/cluster patched
2025-01-09 21:45:16 +(./06_create_cluster.sh:86): [[ -n '' ]]

jianzhangbjz · 2025-01-15T01:48:18Z

Besides, the operatorhub is disabled in many scenarios, such as disconnected and proxy envs, for example, this https://github.com/openshift/release/blob/master/ci-operator/step-registry/ipi/openstack/pre/disconnected/ipi-openstack-pre-disconnected-chain.yaml and https://github.com/openshift/release/blob/14145d08a2d8b367133ab9883bdde96f8443a2f0/ci-operator/step-registry/telco5g/cnf/tests/telco5g-cnf-tests-commands.sh#L451

jianzhangbjz · 2025-01-15T01:50:23Z

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-upgrade-ovn-ipv6

jianzhangbjz · 2025-01-15T01:51:24Z

/test e2e-aws-ovn-microshift-serial

joelanford · 2025-01-15T04:16:52Z

So as soon as a catalog is disabled, the catalog operator will remove the catalogsource_ready time series from its metrics endpoint.

The rule for this alert is defined here: https://github.com/operator-framework/operator-marketplace/blob/5776ce8e796910c2dfc98a42062f97ed98e81b2c/manifests/12_prometheus_rule.yaml#L23

I'm not a prometheus expert, but I think the expression there will just not have results for the time ranges that the default catalogs are disabled. If there are no results, I would think that would mean the expression would not evaluate to true. But I haven't tested this theory.

One possibility to consider. If the catalog source is unhealthy before the catalog source is disabled, the the most recent sample for a given catalogsource_ready time series will be 0. In these clusters where catalog sources are not expected to work (and are thus disabled) it seems likely that they will be unhealthy prior to being disabled, right?

jianzhangbjz · 2025-01-15T06:18:11Z

If there are no results, I would think that would mean the expression would not evaluate to true.

Yes, I think so.

In these clusters where catalog sources are not expected to work (and are thus disabled) it seems likely that they will be unhealthy prior to being disabled, right?

Yes, these catalogsource unhealthy due to the image pulling issue, so they disbaled operatorhub.

jianzhangbjz · 2025-01-15T10:28:23Z

/hold

jan--f · 2025-01-16T14:56:34Z

/lgtm
Imho this here will work to unblock the readiness check. The alerting rule should and/or metric export can be then be fixed.

openshift-ci · 2025-01-16T14:56:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jan--f, jianzhangbjz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/extended/prometheus/OWNERS~~ [jan--f]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jianzhangbjz · 2025-01-17T02:25:40Z

/test e2e-aws-ovn-edge-zones

openshift-ci-robot · 2025-01-17T02:55:20Z

/retest-required

Remaining retests: 0 against base HEAD e356b61 and 2 for PR HEAD 0c45942 in total

openshift-ci-robot · 2025-01-17T05:09:41Z

/retest-required

Remaining retests: 0 against base HEAD e356b61 and 2 for PR HEAD 0c45942 in total

simonpasquier · 2025-01-17T15:15:57Z

The logic is that add OperatorHubSourceError to allowedAlertNames when SkipOperatorHubMetricsCheck is true.

Late to review but i think that it's simpler to exclude OperatorHubSourceError in all tests: the command checking whether OperatorHub is installed could fail or return unexpected results. The root cause for the alert firing despite OperatorHub not installed should be quickly addressed and the exclusion only a temporary counter-measure.

openshift-ci-robot · 2025-01-17T22:09:37Z

/retest-required

Remaining retests: 0 against base HEAD 2d391bf and 1 for PR HEAD 0c45942 in total

openshift-ci-robot · 2025-01-18T05:09:40Z

/retest-required

Remaining retests: 0 against base HEAD 3709116 and 0 for PR HEAD 0c45942 in total

openshift-ci-robot · 2025-01-18T09:09:34Z

/hold

Revision 0c45942 was retested 3 times: holding

jianzhangbjz · 2025-01-20T01:21:12Z

/retest-required
/unhold

openshift-ci-robot · 2025-01-20T06:09:34Z

/retest-required

Remaining retests: 0 against base HEAD a92836d and 2 for PR HEAD 0c45942 in total

jianzhangbjz · 2025-01-20T10:09:02Z

/retest-required

openshift-trt · 2025-01-20T15:09:26Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	Medium [sig-imageregistry][Serial] Image signature workflow can push a signed image to openshift registry and verify it [apigroup:user.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/serial] This test has passed 96.88% of 352 runs on release 4.19 [Overall] in the last week.

openshift-ci-robot · 2025-01-20T15:09:48Z

/retest-required

Remaining retests: 0 against base HEAD 0f2d600 and 1 for PR HEAD 0c45942 in total

openshift-ci-robot · 2025-01-20T19:09:37Z

/retest-required

Remaining retests: 0 against base HEAD 4c63717 and 0 for PR HEAD 0c45942 in total

openshift-trt · 2025-01-20T23:17:45Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	Medium [sig-imageregistry][Serial] Image signature workflow can push a signed image to openshift registry and verify it [apigroup:user.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/serial] This test has passed 96.94% of 359 runs on release 4.19 [Overall] in the last week.

openshift-trt · 2025-01-21T00:09:11Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	Medium [sig-imageregistry][Serial] Image signature workflow can push a signed image to openshift registry and verify it [apigroup:user.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/serial] This test has passed 96.94% of 359 runs on release 4.19 [Overall] in the last week.

openshift-ci-robot · 2025-01-21T00:09:38Z

/hold

Revision 0c45942 was retested 3 times: holding

jianzhangbjz · 2025-01-21T01:36:21Z

/unhold

jianzhangbjz · 2025-01-21T01:43:30Z

/retest-required

openshift-ci-robot · 2025-01-21T06:09:38Z

/retest-required

Remaining retests: 0 against base HEAD 58ddec8 and 2 for PR HEAD 0c45942 in total

jianzhangbjz · 2025-01-21T06:10:48Z

/retest

openshift-trt · 2025-01-21T10:57:42Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-okd-scos-e2e-aws-ovn	High [sig-arch] Only known images used by tests This test has passed 100.00% of 129 runs on jobs [periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn] in the last 14 days.
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade	IncompleteTests Tests for this run (15) are below the historical average (1060): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci-robot · 2025-01-21T14:22:32Z

/retest-required

Remaining retests: 0 against base HEAD 58ddec8 and 2 for PR HEAD 0c45942 in total

openshift-trt · 2025-01-21T18:22:26Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-okd-scos-e2e-aws-ovn	High [sig-arch] Only known images used by tests This test has passed 100.00% of 129 runs on jobs [periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn] in the last 14 days.
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade	IncompleteTests Tests for this run (15) are below the historical average (1107): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	Medium [sig-imageregistry][Serial] Image signature workflow can push a signed image to openshift registry and verify it [apigroup:user.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/serial] This test has passed 97.01% of 401 runs on release 4.19 [Overall] in the last week.

openshift-trt · 2025-01-21T19:08:16Z

Job Failure Risk Analysis for sha: 0c45942

Job Name	Failure Risk
pull-ci-openshift-origin-master-okd-scos-e2e-aws-ovn	High [sig-arch] Only known images used by tests This test has passed 100.00% of 129 runs on jobs [periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn] in the last 14 days.
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade	IncompleteTests Tests for this run (15) are below the historical average (1121): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	Medium [sig-imageregistry][Serial] Image signature workflow can push a signed image to openshift registry and verify it [apigroup:user.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/serial] This test has passed 96.92% of 389 runs on release 4.19 [Overall] in the last week.

openshift-ci-robot · 2025-01-21T19:10:46Z

/retest-required

Remaining retests: 0 against base HEAD a87dfc8 and 2 for PR HEAD 0c45942 in total

openshift-ci · 2025-01-21T23:46:23Z

@jianzhangbjz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-kube-apiserver-rollout	`0c45942`	link	false	`/test e2e-aws-ovn-kube-apiserver-rollout`
ci/prow/e2e-aws-ovn-serial	`0c45942`	link	true	`/test e2e-aws-ovn-serial`
ci/prow/e2e-gcp-ovn-rt-upgrade	`0c45942`	link	false	`/test e2e-gcp-ovn-rt-upgrade`
ci/prow/okd-scos-e2e-aws-ovn	`0c45942`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jan 14, 2025

openshift-ci bot requested review from simonpasquier and slashpai January 14, 2025 07:40

jianzhangbjz force-pushed the metrics branch from a4b5f5f to ba691b5 Compare January 14, 2025 09:19

simonpasquier reviewed Jan 14, 2025

View reviewed changes

skip OperatorHubSourceError metric checking when disableAllDefaultSou…

0c45942

…rces is true

jianzhangbjz force-pushed the metrics branch from ba691b5 to 0c45942 Compare January 14, 2025 09:31

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 15, 2025

openshift-ci bot assigned jan--f Jan 16, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 18, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 20, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2025

	"OperatorHubSourceError", // https://issues.redhat.com/browse/OCPBUGS-48340, should skip this metric checking when disableAllDefaultSources is true
	"OperatorHubSourceError", // https://issues.redhat.com/browse/OCPBUGS-48340

OCPBUGS-48340: skip OperatorHubSourceError metric checking when disableAllDefaultSources is true #29435

Are you sure you want to change the base?

OCPBUGS-48340: skip OperatorHubSourceError metric checking when disableAllDefaultSources is true #29435

Conversation

jianzhangbjz commented Jan 14, 2025

openshift-ci-robot commented Jan 14, 2025

jianzhangbjz commented Jan 14, 2025

simonpasquier commented Jan 14, 2025

jianzhangbjz commented Jan 14, 2025

jianzhangbjz commented Jan 14, 2025

simonpasquier Jan 14, 2025

Choose a reason for hiding this comment

simonpasquier Jan 14, 2025

Choose a reason for hiding this comment

simonpasquier Jan 14, 2025

Choose a reason for hiding this comment

jianzhangbjz commented Jan 14, 2025

joelanford commented Jan 14, 2025

jianzhangbjz commented Jan 15, 2025

jianzhangbjz commented Jan 15, 2025

jianzhangbjz commented Jan 15, 2025

jianzhangbjz commented Jan 15, 2025

joelanford commented Jan 15, 2025

jianzhangbjz commented Jan 15, 2025 • edited Loading

jianzhangbjz commented Jan 15, 2025

jan--f commented Jan 16, 2025

openshift-ci bot commented Jan 16, 2025

jianzhangbjz commented Jan 17, 2025

openshift-ci-robot commented Jan 17, 2025

openshift-ci-robot commented Jan 17, 2025

simonpasquier commented Jan 17, 2025

openshift-ci-robot commented Jan 17, 2025

openshift-ci-robot commented Jan 18, 2025

openshift-ci-robot commented Jan 18, 2025

jianzhangbjz commented Jan 20, 2025

openshift-ci-robot commented Jan 20, 2025

jianzhangbjz commented Jan 20, 2025

openshift-trt bot commented Jan 20, 2025

openshift-ci-robot commented Jan 20, 2025

openshift-ci-robot commented Jan 20, 2025

openshift-trt bot commented Jan 20, 2025

openshift-trt bot commented Jan 21, 2025

openshift-ci-robot commented Jan 21, 2025

jianzhangbjz commented Jan 21, 2025

jianzhangbjz commented Jan 21, 2025

openshift-ci-robot commented Jan 21, 2025

jianzhangbjz commented Jan 21, 2025

openshift-trt bot commented Jan 21, 2025

openshift-ci-robot commented Jan 21, 2025

openshift-trt bot commented Jan 21, 2025

openshift-trt bot commented Jan 21, 2025

openshift-ci-robot commented Jan 21, 2025

openshift-ci bot commented Jan 21, 2025

jianzhangbjz commented Jan 15, 2025 •

edited

Loading