4.16.0-okd-scos.1 cannot be verified against keyrings

[ibotty]

When trying to update I get

Cluster version is 4.16.0-okd-scos.0

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="4.16.0-okd-scos.1" image="registry.ci.openshift.org/origin/release-scos@sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1" failure=The update cannot be verified: unable to verify sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1 against keyrings: verifier-public-key-redhat

Upstream: https://origin-release.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

  VERSION           IMAGE
  4.16.0-okd-scos.1 registry.ci.openshift.org/origin/release-scos@sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1

This cluster used to be a FCOS based OKD.

[f5-jay]

I'm observing the same error.

[madpearl]

Hi,
just do:

 oc adm upgrade --clear
 ## Wait till upgrade is canceled
 ## run 
 oc adm upgrade --to-image=quay.io/okd/scos-release:4.16.0-okd-scos.1 --force --allow-explicit-upgrade

[BeardOverflow]

It is a common error due to signed payload key used when you are mixing nightly/rc/ec/ga releases. Go to page 2 here https://mirror.openshift.com/pub/openshift-v4/OpenShift_Release_Types.pdf

[ibotty]

This was a cluster that had been an OKD-FCOS, but the release before updating to 4.16.0-okd-scos.1 was 4.16.0-okd-scos.0.

Just from another cluster:

$ oc adm upgrade
Cluster version is 4.16.0-okd-scos.0

ReleaseAccepted=False

  Reason: RetrievePayload
  Message: Retrieving payload failed version="4.16.0-okd-scos.1" image="registry.ci.openshift.org/origin/release-scos@sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1" failure=The update cannot be verified: unable to verify sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1 against keyrings: verifier-public-key-redhat

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

  VERSION           IMAGE
  4.16.0-okd-scos.1 registry.ci.openshift.org/origin/release-scos@sha256:06ffff6c6951046d03df0784bc18132c368a84fe72bcfb529484a58872c3a2e1

[BeardOverflow]

Uhm, looks good.

Please, can you review this configmap?

oc get -n openshift-config-managed configmap/release-verification -o yaml

And verify the content for the following keys:

• store-openshift-ci-release
• verifier-public-key-openshift-ci

oc adm upgrade uses the above keys.

[ibotty]

store-openshift-ci-release is the same.
I don't have a verifier-public-key-openshift-ci key but only a verifier-public-key-ci and verifier-public-key-redhat, both of them are different.

This specific cluster had first been installed as 4.11.0-0.okd-2022-07-29-154152 AFAICT.

The other cluster that failed to verify has the same configmap and was installed as 4.9.0-0.okd-2021-11-28-035710.

[BeardOverflow]

Try to change the values in the configmap manually (OKD uses openshift-ci PGP keys/GCP endpoints) and re-launch the upgrade.

I think that it could be a bug in the cluster-update-keys/cluster-version-operator.