CentOS Stream 10 build is failing with error symvers-*.ppc64le.gz does not exist in the boot directory

[Poorna-Gottimukkula1]

What happened:

CentOS Stream 10 build is failing with below error.

[root@ocp4alll367 2025-Jan-24]# pvsadm image qcow2ova --image-name centos-10-stream-20250122  --image-dist centos  --image-url ./CentOS-Stream-GenericCloud-10-20250122.0.ppc64le.qcow2
I0124 04:07:02.716753   39477 qcow2ova.go:178] Autogenerated OS root password is updated in /root/builds/2025-Jan-24/password.txt
I0124 04:07:02.716845   39477 validate.go:40] Checking: platform
I0124 04:07:02.716862   39477 validate.go:40] Checking: user
I0124 04:07:02.716872   39477 validate.go:40] Checking: image-name
I0124 04:07:02.717212   39477 validate.go:40] Checking: tools
I0124 04:07:02.717323   39477 tools.go:43] qemu-img found at /usr/bin/qemu-img
I0124 04:07:02.717377   39477 tools.go:43] growpart found at /usr/bin/growpart
I0124 04:07:02.717389   39477 validate.go:40] Checking: diskspace
I0124 04:07:02.717411   39477 diskspace.go:49] free: 113G, need: 61G
I0124 04:07:02.725351   39477 qcow2ova.go:251] Converting Qcow2(/tmp/qcow2ova2360822010/CentOS-Stream-GenericCloud-10-20250122.0.ppc64le.qcow2) image to raw(/tmp/qcow2ova2360822010/ova-img-dir/centos-10-stream-20250122-disk.raw) format
I0124 04:07:06.931698   39477 qcow2ova.go:256] Conversion completed
I0124 04:07:06.931724   39477 qcow2ova.go:258] Resizing the image /tmp/qcow2ova2360822010/ova-img-dir/centos-10-stream-20250122-disk.raw to 11G
Image resized.
I0124 04:07:13.108074   39477 qcow2ova.go:263] Resize completed
I0124 04:07:13.108099   39477 qcow2ova.go:265] Preparing the image
/dev/loop0
Warning: Not all of the space available to /dev/loop0 appears to be used, you can fix the GPT to use all of the space (an extra 2097152 blocks) or continue with the current setting?
GPT PMBR size mismatch (20971519 != 23068671) will be corrected by write.
The backup GPT table is not on the end of the device.
2
CHANGED: partition=2 start=10240 old: size=16384000 end=16394239 new: size=23058399 end=23068638
xfs
meta-data=/dev/loop0p2           isize=512    agcount=4, agsize=512000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=0 inobtcount=0 nrext64=1
data     =                       bsize=4096   blocks=2048000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2048000 to 2882299
Error: failed while preparing the image for centos distro, err: symvers-*.ppc64le.gz does not exist in the boot directory
Usage:
  pvsadm image qcow2ova [flags]

Flags:
      --image-name string       Name of the resultant OVA image
      --image-url string        URL or absolute local file path to the <QCOW2>.gz image
      --image-dist string       Image Distribution(supported: rhel, centos, coreos)
      --image-size uint         Size (in GB) of the resultant OVA image (default 11)
      --target-disk-size int    Size (in GB) of the target disk volume where OVA will be copied (default 120)
      --rhn-user string         RedHat Subscription username. Required when Image distribution is rhel
      --rhn-password string     RedHat Subscription password. Required when Image distribution is rhel
      --os-password string      Root user password, will auto-generate the 12 bits password(applicable only for redhat and cento distro)
  -t, --temp-dir string         Scratch space to use for OVA generation (default "/tmp")
      --prep-template string    Image preparation script template, use --prep-template-default to print the default template(supported distros: rhel and centos)
      --prep-template-default   Prints the default image preparation script template, use --prep-template to set the custom template script(supported distros: rhel and centos)
      --skip-os-password        Skip the root user password
      --cloud-config string     Set the custom cloud config, use --cloud-config-default to print the default cloud config
      --cloud-config-default    Prints the default cloud config template, use --cloud-config to set the custom cloud config template
  -h, --help                    help for qcow2ova

Global Flags:
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --alsologtostderr                  log to standard error as well as files (no effect when -logtostderr=true)
  -k, --api-key string                   IBMCLOUD API Key(env name: IBMCLOUD_APIKEY)
      --audit-file string                Audit logs for the tool (default "pvsadm_audit.log")
      --debug                            Enable PowerVS debug option(ATTENTION: dev only option, may print sensitive data from APIs)
      --env string                       IBM Cloud Environments, supported are: [test, prod] (default "prod")
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory (no effect when -logtostderr=true)
      --log_file string                  If non-empty, use this log file (no effect when -logtostderr=true)
      --log_file_max_size uint           Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
      --one_output                       If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files (no effect when -logtostderr=true)
      --stderrthreshold severity         logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=true) (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

E0124 04:07:13.537359   39477 main.go:26] failed while preparing the image for centos distro, err: symvers-*.ppc64le.gz does not exist in the boot directory
[root@ocp4alll367 2025-Jan-24]#

What you expected to happen:

Image build should be successfull

How to reproduce it (as minimally and precisely as possible):

Download latest available cloud image from here https://cloud.centos.org/centos/10-stream/ppc64le/images/CentOS-Stream-GenericCloud-10-latest.ppc64le.qcow2

# wget https://cloud.centos.org/centos/10-stream/ppc64le/images/CentOS-Stream-GenericCloud-10-20250122.0.ppc64le.qcow2
# pvsadm image qcow2ova --image-name centos-10-stream-20250122 --image-dist centos  --image-url ./CentOS-Stream-GenericCloud-10-20250122.0.ppc64le.qcow2

Anything else we need to know?:

Environment:

• pvsadm version (use pvsadm --version): Version: v0.1.20 GoVersion: go1.22.4
• Environment - On-Prem or IBM PowerVS: on prem
• OS (e.g: cat /etc/os-release):

NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

• Kernel (e.g. uname -a):

Linux image-build-centos-stream-tests.pokprv.stglabs.ibm.com 5.14.0-312.el9.ppc64le #1 SMP Thu May 11 16:13:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux

• Others:

[KeerthanaAP]

/assign

[Poorna-Gottimukkula1]

When tested with an older version of pvsadm (Version: v0.1.14, GoVersion: go1.21.6), it progressed further, and the symvers-*.ppc64le* file in the boot directory had a .xz extension.

#  ls -la /
total 111972
dr-xr-xr-x.  5 root root     4096 Jan 23 11:49 .
dr-xr-xr-x. 18 root root      251 Jan 24 11:18 ..
-rw-r--r--.  1 root root   180778 Jan 19 19:00 config-6.12.0-43.el10.ppc64le
drwxr-xr-x.  3 root root       17 Jan 23 11:47 efi
drwx------.  5 root root      118 Jan 23 11:49 grub2
-rw-------.  1 root root 60975142 Jan 23 11:50 initramfs-6.12.0-43.el10.ppc64le.img
drwxr-xr-x.  3 root root       21 Jan 23 11:48 loader
-rw-r--r--.  1 root root   211640 Jan 23 11:49 symvers-6.12.0-43.el10.ppc64le.xz
-rw-r--r--.  1 root root  5180118 Jan 19 19:00 System.map-6.12.0-43.el10.ppc64le
-rwxr-xr-x.  1 root root 48095024 Jan 19 19:00 vmlinuz-6.12.0-43.el10.ppc64le
-rw-r--r--.  1 root root      161 Jan 19 19:00 .vmlinuz-6.12.0-43.el10.ppc64le.hmac

Its seems the latest version of pvsadm had checks for symvers-*.ppc64le.gz, and it failed to locate the corresponding file in the boot directory.

However, it is still failing at some point that its not able to install the ibm power packages.

Retrieving https://public.dhe.ibm.com/software/server/POWER/Linux/yum/download/ibm-power-repo-latest.noarch.rpm
Verifying...                          ########################################
Preparing...                          ########################################
Updating / installing...
ibm-power-repo-3.0.0-22               ########################################

Licensing information

In addition to the GNU Public License (GPL), each of these packages
includes its own licensing information. Please consult the individual
tools for their licensing information.

The freeware packages ("Code") provided here are made available as a
convenience to IBM customers. IBM does not own the Code, and did not
develop or exhaustively test the Code. IBM does not provide support
for the Code.  These packages are available for download via
http://www-304.ibm.com/webapp/set2/sas/f/lopdiags/yum.html.

Each Code package is subject to the terms of the license that is
included in the Code package. By downloading a particular package you
agree to accept and comply with the license terms that accompany such
package.

No Warranty: The Code is provided "As is." To the extent permitted by
applicable law, IBM disclaims all warranties either express or
implied, including without limitation any warranty of
non-infringement, noninterference, merchantability, or fitness for a
particular purpose regarding the code or technical support, if any.

This exclusion also applies to any of IBM's subcontractors, suppliers,
or program developers (collectively called "Suppliers"). Limitation of
Liability: Neither IBM nor its suppliers are liable for any direct or
indirect damages, including without limitation, lost profits, lost
savings, or any incidental, special, or other economic consequential
damages, even if IBM is informed of their possibility. Some
jurisdictions do not allow the exclusion or limitation of incidental
or consequential damages, so the above exclusion or limitation may not
apply to you.

Introduction

A number of open source packages have been prepared for installation
on Linux using the RPM Package Manager. These packages, also known as
RPMs, are available on an as-is basis for beta-level use.  Once you
accept these terms, you will be granted access to RPMs that will allow
you to access the packages.  The RPMs are deemed IBM confidential
information.  You will not (i) disclose, distribute, transfer or
otherwise provide RPMs to any third party, (ii) use RPMs outside of
your enterprise or (iii) remove any copyright or other notices from
RPMs.  You will ensure that anyone within your enterprise who uses
RPMs does so only on your behalf and complies with these terms.  If
you do not comply with these terms, IBM may terminate your rights to
access the packages.

Go here
http://www-304.ibm.com/webapp/set2/sas/f/lopdiags/toolsupport.html for
a listing of tools.


Adding repo from: https://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/10/ppc64le/
curl: (22) The requested URL returned error: 404
error: https://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/10/ppc64le/repodata/repomd.xml.key: import read failed(2).
Error: failed while preparing the image for centos distro, err: script /setup.sh failed with exitstatus: 1, stdout: nameserver 9.9.9.9

Even I tried changing to RHEL 9 version https://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/9/ppc64le/ and I'm still hitting the below error.

Go here
http://www-304.ibm.com/webapp/set2/sas/f/lopdiags/toolsupport.html for
a listing of tools.


Adding repo from: https://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/9/ppc64le/
, stderr: Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <[email protected]>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial-SHA256
error: Certificate 0A2858B03E6E42BE:
  Policy rejects 0A2858B03E6E42BE: No binding signature at time 2025-01-24T16:19:25Z
error: https://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/9/ppc64le/repodata/repomd.xml.key: key 1 import failed.