Apply unmerged patches from upstream unmaintained/yoga #83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The job definition does not exist anymore and zuul config reports error because of this, so let's get rid of this from the check queue (gate queue does not even have it). Change-Id: I0b211b7c8fcfebeaf9f1bc9bbee0c6527faa53d9
py39 jobs (on ubuntu-focal) started to fail due to recent virtualenv release (20.26.4 that bundles setuptools 74.1.2) on Yoga, because we have 'packaging==21.3' in this branch that is not compatible with the new setuptools [1]. setuptools is bundled in virtualenv, so it has to be capped via the virtualenv package. tox also needed to be capped (<4) as gate uses tox 3.28.0, but with capping virtualenv we pull in latest tox as well, which would cause other errors. [1] pypa/setuptools#4483 Change-Id: I668557933c51123ce84275a0c718b6e79b3df174
This adds a safety check and detection mechanism to the tools/test_format_inspector.py utility for verifying those features outside of glance. Change-Id: I447e7e51315472f8fa6013d4c4852f54c1e0c43d (cherry picked from commit b27040b87e43ff4acfb6870ef0d13d54e5ca5caa) (cherry picked from commit b394ef00c3426771092d099cf1d96077bfa4b919) (cherry picked from commit aa12d39b453068d9ee8367591eb60d4a15cddece) (cherry picked from commit a1c94b4) (cherry picked from commit a5ec69b)
We have to address two issues simultaneously: 1. handle multiple cirros images The change[1] configures multiple images for the devstack-tempest job and all the jobs deriving from it. This causes problem with the current post-check-metadata-injection.yaml playbook since we only expect one cirros image and perform steps accordingly. This change modifies the playbook to handle multiple cirros images. Failure in the glance-multistore-cinder-import job can be seen here[2]. [1] https://review.opendev.org/c/openstack/tempest/+/831018 [2] https://zuul.opendev.org/t/openstack/build/68a4d3ec6ce04c87b21d73a333f5b5cd/log/job-output.txt#23161 This was change I70b4a970d7e1. 2. change the the job config to address resource constraints * partial backport of I073216d1bbdd to change swap size (we don't include the change from that patch that was needed to address changes in Bobcat-era devstack) * backporting Ieb96eb6ceb6f to change tempest concurrency Remove openstack-tox-functional-py36-fips as there is no job definition for it. Change-Id: I6cd5b8bbf39f0cfae466f0800faea5a74e960e69 (cherry picked from commit 065e2d7) (cherry picked from commit 28b05a7)
VMDK: When parsing a VMDK file to calculate its size, the format_inspector determines the location of the Descriptor section by reading two uint64 from the headers of the file and uses them to create the descriptor CaptureRegion. It would be possible to craft a VMDK file that commands the format_inspector to create a very big CaptureRegion, thus exhausting resources on the glance-api process. This patch binds the beginning of the descriptor to 0x200 and limits the size of the CaptureRegion to 1MB, similar to how the VMDK descriptor is parsed by qemu. VHDX: It is a bit more involved, but similar: when looking for the VIRTUAL_DISK_SIZE metadata, the format_inspector was creating an unbounded CaptureRegion. In the same way as it seems to be done in Qemu, we now limit the upper bound of this CaptureRegion. Closes-Bug: #2006490 Change-Id: I3ec5a33df20e1cfb6673f4ff1c7c91aacd065532 (cherry picked from commit d4d33ee) (cherry picked from commit 06a1820)
Stream optimized VMDKs are also monolithic disks images, and start with the same sparse extend header as normal monolithic sparse files, so we can parse the virtual disk size in the same manner. See "VMware Virtual Disks Virtual Disk Format 1.1" p. 17. > Header and Footer > The header and the footer are both described by the same SparseExtentHeader > structure shown in Hosted Sparse Extent Header on page 8. Closes-Bug: #2052291 Change-Id: I7d63951ff080dc699b8d11babc0a5998d90621e4 Co-Authored-By: Rajiv Mucheli <[email protected]> (cherry picked from commit 5e7e6bf) (cherry picked from commit 9d45e8d4b87e992be23974a831811bff563ce721) (cherry picked from commit ef22a74) (cherry picked from commit bcfc1e4)
Change-Id: I6326a3e85c1ba4cb1da944a4323769f2399ed2c1 Closes-Bug: #2059809 (cherry picked from commit 2ca29af4433e9fa99a0a48e230d8d25d6eaa4a87) (cherry picked from commit c3586f3a122f6cb0663217b12b52203e74e2e4fa) (cherry picked from commit a92c438fb5ba55440b38cae7c8b4361b58daa9dd) (cherry picked from commit dba3bdb) (cherry picked from commit 6a38aef)
This adds two properties to the QcowInspector that makes it able to indicate whether the file specifies a backing_file or data_file in the header. Both conditions are considered unsafe for our usage. To ease checking of this condition, a classmethod is added that takes a local filename and digests just enough of the file to assert that both conditions are false. Note for yoga backport:With older qemu installed one of the qemu-img create commands fails, let's skip it from unmaintained/yoga and below that. Change-Id: Iaf86b525397d41bd116999cabe0954a0a7efac65 Related-Bug: #2059809 (cherry picked from commit ae536bb394793c9a7a219cb498e03d5c81dbbbb7) (cherry picked from commit 2eba54e0821106097dfeceb424e53943fd090483) (cherry picked from commit 89dbbc838d606f461087e1494d19ddbcf9db0a38) (cherry picked from commit 4860024) (cherry picked from commit f32d5b8)
This makes us check the extent filenames to make sure they don't have any banned characters in them (i.e. slashes). It also makes us reject VMDK files with a footer. Since we process these files as a stream, we can't honor a footer that directs us to find the descriptor block in a location we've already processed. Thus, if a file indicates it has a footer, consider it a policy exception and unsupported. Change-Id: I4a1c6dff7854c49940a0ac7988722aa6befc04fa (cherry picked from commit c1bf35dffb7f4c3090b1f04cf0e27cb431330c3e) (cherry picked from commit d3f1d6159c0218ac01e8d881e2ec4da71fc952ee) (cherry picked from commit 2dd4d138d4b8e1d9ca69fc0dda3711553a65d912) (cherry picked from commit 812e56d) (cherry picked from commit 05650e1)
Recent changes does not allow image conversion to convert an image having AMI disk format for image. Passing list of disk-formats to each import job which will now use qcow2 as default disk-format to create the image. Conflicts: .zuul.yaml Change-Id: I712b304ebe168705ec2793626f5d64135ad6d458 (cherry picked from commit 244ccb6) (cherry picked from commit f98e9ae) (cherry picked from commit 02732c2)
This patch contains multiple fixes to unblock the gate:
1. use appropriate version of tempest repos
glance-secure-rbac-protection-functional job is failing as it uses too
recent version of tempest plugin which is not compatible with yoga
branch anymore [1]. This patch uses yoga-last tagged tempest and plugin
for the job.
2. drop centos-8 based job
glance-multistore-cinder-import-fips used centos-8 based nodeset which
was dropped from CI config (due to centos-8 reached End of Life)
[1] "ERROR: Package 'glance-tempest-plugin' requires a different
Python: 3.8.10 not in '>=3.9'"
Change-Id: I1fd66444bfb313969c45e77fed4daf8e5d0fc9f4
This causes us to use the format inspector to pre-examine qcow and vmdk files for safe configurations before even using qemu-img on them. Conflict: test_image_conversion.py Conflict: api_image_import.py has propery image_disk_format in later branches, that is necessary to make imge converter methods work. From Ic51c5fd87caf04d38aeaf758ad2d0e2f28098e4d Change-Id: I0554706368e573e11f649c09569f7c21cbc8634b Closes-Bug: #2059809 (cherry picked from commit a95f335bca1dfdd1c904ae475e9fe8c6806f2c56) (cherry picked from commit 55fc42563818fcf88b474233df242a796c918b3a) (cherry picked from commit f1f53075d69a9a1c006b3e25506e30eb0210de1f) (cherry picked from commit c1c54ab) (cherry picked from commit 63ac35e)
This merely recognizes this format and always marks it as unsafe because no service supports it. This prevents someone from uploading one that we will ask qemu-img to inspect. Change-Id: Ieea7b7eb0f380571bd4937cded920776e05f7ec4 (cherry picked from commit 4096c5aff1d046d5c28d0e4a69b3c9574e9e8cc8) (cherry picked from commit ba98022b98ef5b9c98e1d7d20c88e1ca4b23fa80) (cherry picked from commit a8dadcd7994c88a61f94341286b4bcd693b51b32) (cherry picked from commit 9b3faf3)
Change-Id: If0a4251465507be035ffaf9d855299611637cfa9 (cherry picked from commit 79271eaa5c742a1741321198c43807857fb6ed94) (cherry picked from commit e1c36248c7660dea1bedfa8f1c0711a4b97d279c) (cherry picked from commit d54121d6a937fd50aae1018aede228a3c0985dce) (cherry picked from commit 1656fe8)
Change-Id: I8eebc1d6019047b5ab6f60e1057c0dd81c31cb6f
Change-Id: Id3578d6137cc5c30986839eadbb5c85062e5ee75
dougszumski
approved these changes
Jan 29, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Upstream unmaintained/yoga has several security patches not yet merged because of CI issues. This change applies all these patches, which should also fix the weekly sync jobs.