Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(arm,bootloaders): use correct blsdir when boot is not separate #1339

Merged
merged 1 commit into from
Feb 8, 2025

Conversation

MichalHe
Copy link
Member

@MichalHe MichalHe commented Feb 6, 2025

Currently, leapp does not distinguish between having and not having a separate /boot partition. Since blsdir is relative to the device on which /boot resides, ARM systems without a separate /boot could not boot into our upgrade environment, and, instead, they booted back into pre-upgrade RHEL8. This patch adds additional handling for non-separate /boot to the process of specifying custom blsdir.

Resolves: RHEL-41193

Copy link

github-actions bot commented Feb 6, 2025

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

  • review please @oamg/developers to notify leapp developers of the review request
  • /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build.
However, here are additional useful commands for packit:

  • /packit test to re-run manually the default tests
  • /packit retest-failed to re-run failed tests manually
  • /packit test oamg/leapp#42 to run tests with leapp builds for the leapp PR#42 (default is latest upstream - main - build)

Note that first time contributors cannot run tests automatically - they need to be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

  • /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
  • /packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

@MichalHe MichalHe force-pushed the bootloaders_no_sep_boot branch 3 times, most recently from 1eaf960 to 0ee4984 Compare February 7, 2025 21:35
Currently, leapp does not distinguish between having and not having a
separate /boot partition. Since blsdir is relative to the device
on which /boot resides, ARM systems without a separate /boot could
not boot into our upgrade environment, and, instead, they booted
back into pre-upgrade RHEL8. This patch adds additional handling
for non-separate /boot to the process of specifying custom blsdir.

Resolves: RHEL-41193
@MichalHe MichalHe force-pushed the bootloaders_no_sep_boot branch from 0ee4984 to 9e25c17 Compare February 7, 2025 21:55
@MichalHe
Copy link
Member Author

MichalHe commented Feb 8, 2025

Logs from upgrading a system (8.10 > 9.5) without a separate /boot

System's disk setup:

[root@ip-172-31-36-66 ~]# lsblk -o NAME,UUID,MOUNTPOINT
NAME        UUID                                 MOUNTPOINT
nvme0n1
├─nvme0n1p1 7B77-95E7                            /boot/efi
├─nvme0n1p2 50cc8d0c-6cae-4993-aa38-5b7209dce8ce
└─nvme0n1p3 0f5e676b-368d-4357-921b-f2dd60a9109a /

After running leapp upgrade --debug --no-rhsm --target 9.5

Leapp's EFI entry is set to be used in the next boot:

[root@ip-172-31-36-66 ~]# efibootmgr -v
BootNext: 0003
BootCurrent: 0002
Timeout: 0 seconds
BootOrder: 0003,0002,0000,0001
Boot0000* UiApp FvVol(64074afe-340a-4be6-94ba-91b5b4d0f71e)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI Amazon Elastic Block Store vol015acde32e5df6d96 1        PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot0002* Red Hat Enterprise Linux      HD(1,GPT,68b2905b-df3e-4fb3-80fa-49d1e773aa33,0x800,0x64000)/File(\EFI\redhat\shimaa64.efi)
Boot0003* Leapp Upgrade HD(1,GPT,68b2905b-df3e-4fb3-80fa-49d1e773aa33,0x800,0x64000)/File(\EFI\leapp\shimaa64.efi)

Leapp's grub.cfg consists of the following. Note that grub's root is set to correct UUID.

[root@ip-172-31-36-66 ~]# cat /boot/efi/EFI/leapp/grub.cfg
set timeout=0

# Make sure to load EFI/leapp/grubenv and not system's default path
if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi

# EFI/leapp/grubenv contains our upgrade BLS entry as saved_entry
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

search --no-floppy --set=root --fs-uuid 0f5e676b-368d-4357-921b-f2dd60a9109a
set boot=${root}
function load_video {
  insmod all_video
}
${serial}${terminal_input}${terminal_output}
blscfg

Leapp's grubenv contains blsdir set correctly, that is blsdir is a full path from / of the system. Note that saved_entry plays no role and it will be ignored since it cannot be found in the available entry list.

[root@ip-172-31-36-66 ~]# cat /boot/efi/EFI/leapp/grubenv
# GRUB Environment Block
kernelopts=root=UUID=0f5e676b-368d-4357-921b-f2dd60a9109a console=tty0 console=ttyS0,115200n8 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 iommu.strict=0 crashkernel=auto
saved_entry=ffffffffffffffffffffffffffffffff-4.18.0-553.5.1.el8_10.aarch64
boot_success=1
blsdir=/boot/upgrade-loader
################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################[root@ip-172-31-36-66 ~]#

Leapp's blsdir contains only one boot entry:

[root@ip-172-31-36-66 ~]# ls /boot/upgrade-loader/
ec212eb899e0458144be76d64935684c-upgrade.aarch64.conf

containing

[root@ip-172-31-36-66 ~]# cat /boot/upgrade-loader/ec212eb899e0458144be76d64935684c-upgrade.aarch64.conf
title RHEL-Upgrade-Initramfs
version upgrade.aarch64
linux /boot/vmlinuz-upgrade.aarch64
initrd /boot/initramfs-upgrade.aarch64.img
options $kernelopts $tuned_params enforcing=0 plymouth.enable=0 rd.plymouth=0
id rhel-0-upgrade.aarch64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

Note that both linux and initrd are have the prefix /boot which is correct and required.

Hitting reboot, the system boots into grub menu with only one (selected entry). Note that I have increased the timeout to capture this information.


                               GRUB version 2.06

 /----------------------------------------------------------------------------\
 |*RHEL-Upgrade-Initramfs                                                     | 
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            | 
 \----------------------------------------------------------------------------/

      Use the ^ and v keys to select which entry is highlighted.          
      Press enter to boot the selected OS, `e' to edit the commands       
      before booting or `c' for a command-line.                           

The entry boots into the upgrade environment and the upgrade proceeds.

@MichalHe
Copy link
Member Author

MichalHe commented Feb 8, 2025

Logs from upgrading a system (8.10 > 9.5) with a separate /boot

System's disk setup:

[root@ip-172-31-36-185 ~]# lsblk -o NAME,UUID,MOUNTPOINT
NAME        UUID                                 MOUNTPOINT
nvme0n1
├─nvme0n1p1 7B77-95E7                            /boot/efi
├─nvme0n1p2 50cc8d0c-6cae-4993-aa38-5b7209dce8ce /boot
└─nvme0n1p3 0f5e676b-368d-4357-921b-f2dd60a9109a /

After running leapp upgrade --debug --no-rhsm --target 9.5

Leapp's EFI entry is set to be used in the next boot:

[root@ip-172-31-36-185 ~]# efibootmgr -v
BootNext: 0003
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0003,0002,0000,0001
Boot0000* UiApp	FvVol(64074afe-340a-4be6-94ba-91b5b4d0f71e)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI Amazon Elastic Block Store vol0c64df3a9b2dfe989 1	PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,00-00-00-00-00-00-00-00)N.....YM....R,Y.
Boot0002* Red Hat Enterprise Linux	HD(1,GPT,68b2905b-df3e-4fb3-80fa-49d1e773aa33,0x800,0x64000)/File(\EFI\redhat\shimaa64.efi)
Boot0003* Leapp Upgrade	HD(1,GPT,68b2905b-df3e-4fb3-80fa-49d1e773aa33,0x800,0x64000)/File(\EFI\leapp\shimaa64.efi)

Leapp's grub.cfg consists of the following. Note that grub's root is set to correct UUID of the disk containing /boot.

[root@ip-172-31-36-185 ~]# cat /boot/efi/EFI/leapp/grub.cfg
set timeout=0

# Make sure to load EFI/leapp/grubenv and not system's default path
if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi

# EFI/leapp/grubenv contains our upgrade BLS entry as saved_entry
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

search --no-floppy --set=root --fs-uuid 50cc8d0c-6cae-4993-aa38-5b7209dce8ce
set boot=${root}
function load_video {
  insmod all_video
}
${serial}${terminal_input}${terminal_output}
blscfg

Leapp's grubenv contains blsdir set correctly, that is blsdir is a relative path from /boot . Note that saved_entry plays no role and it will be ignored since it cannot be found in the available entry list.

[root@ip-172-31-36-185 ~]# cat /boot/efi/EFI/leapp/grubenv
# GRUB Environment Block
kernelopts=root=UUID=0f5e676b-368d-4357-921b-f2dd60a9109a console=tty0 console=ttyS0,115200n8 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 iommu.strict=0 crashkernel=auto
saved_entry=ffffffffffffffffffffffffffffffff-4.18.0-553.5.1.el8_10.aarch64
boot_success=1
blsdir=/upgrade-loader
#####################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################[root@ip-172-31-36-185 ~]#

Leapp's blsdir contains only one boot entry:

[root@ip-172-31-36-185 ~]# ls /boot/upgrade-loader/
ec27a79b94ee04151a4e862414c21954-upgrade.aarch64.conf

containing

[root@ip-172-31-36-185 ~]# cat /boot/upgrade-loader/ec27a79b94ee04151a4e862414c21954-upgrade.aarch64.conf
title RHEL-Upgrade-Initramfs
version upgrade.aarch64
linux /vmlinuz-upgrade.aarch64
initrd /initramfs-upgrade.aarch64.img
options $kernelopts $tuned_params enforcing=0 plymouth.enable=0 rd.plymouth=0
id rhel-0-upgrade.aarch64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

Note that the kernel and initramfs are relative to /boot, which is correct.

Hitting reboot, the system boots into grub menu with only one (selected entry). Note that I have increased the timeout to capture this information.


                               GRUB version 2.06

 /----------------------------------------------------------------------------\
 |*RHEL-Upgrade-Initramfs                                                     | 
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            | 
 \----------------------------------------------------------------------------/

      Use the ^ and v keys to select which entry is highlighted.          
      Press enter to boot the selected OS, `e' to edit the commands       
      before booting or `c' for a command-line.                           

Hitting e on the entry shows:


                               GRUB version 2.06

 /----------------------------------------------------------------------------\
 |load_video                                                                  | 
 |set gfxpayload=keep                                                         |
 |insmod gzio                                                                 |
 |linux ($root)/vmlinuz-upgrade.aarch64 root=UUID=0f5e676b-368d-4357-921b-f2d\|
 |d60a9109a console=tty0 console=ttyS0,115200n8 net.ifnames=0 rd.blacklist=no\|
 |uveau nvme_core.io_timeout=4294967295 iommu.strict=0 crashkernel=auto enfor\|
 |cing=0 plymouth.enable=0 rd.plymouth=0                                      |
 |initrd ($root)/initramfs-upgrade.aarch64.img                                |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            | 
 \----------------------------------------------------------------------------/

      Minimum Emacs-like screen editing is supported. TAB lists           
      completions. Press Ctrl-x or F10 to boot, Ctrl-c or F2 for a        
      command-line or ESC to discard edits and return to the GRUB menu.   

Hitting ESC and Enter enters the upgrade environment and proceeds with the upgrade.

Copy link
Member

@pirat89 pirat89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm and based on logs in comments above it works. Thanks!! @MichalHe

@pirat89 pirat89 merged commit 737b3b4 into oamg:main Feb 8, 2025
23 checks passed
@pirat89 pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Feb 8, 2025
@pirat89 pirat89 added this to the 8.10/9.6 milestone Feb 8, 2025
pirat89 added a commit to pirat89/leapp-repository that referenced this pull request Feb 14, 2025
## Announce
This is the last upstream release covering IPU 7 -> 8. The
system_upgrade_el7toel8 repository is going to be removed from further
releases. Other artifacts related to IPU 7 -> 8 are going to be removed too.

In the same time it's the last release compatible with Python 2.7. Since now
we are going to develope and test the code only for Python 3.6 and newer.

## Packaging
- [IPU 9 -> 10] Require libdb-utils on RHEL 9 (oamg#1289)
- Require leapp-framework 6.0+ (oamg#1142)
- Update leapp-deps package to satisfy leapp_framework_deps 6 (oamg#1142)

## Upgrade handling
### Fixes
- Activate LVM VGs with `--sysinit` option to correct the use in the upgrade initramfs (oamg#1329)
- Cap max size of the sparse files to 1TiB when storage with large amount of space (oamg#1294)
- Fix IPU being blocked by resource limitations (oamg#1256)
- Fix pes events scanner crashing when there are duplicate packages in the received instructions (oamg#1296)
- Fix pes events scanner not respecting user’s transaction configuration (oamg#1296)
- Fix storage scanner crashing when command outputs contain colon character (oamg#1258)
- Fix the report when handling broken parsing of kernel cmdline (oamg#1290)
- Generate proper error message instead ModelViolationError when parsing invalid repository definition (oamg#1266)
- Handle default kernel cmdline when multiple boot entries for the default kernel are defined (oamg#1302)
- Load all available obsoleted keys (oamg#1286)
- Prevent a possible crash with LiveMode when adding the upgrade boot entry on LVM systems (oamg#1313)
- Prevent failure for upgrade paths when obsoleted keys are not defined (oamg#1285)
- [IPU 7 -> 8] Ignore invalid firewalld configuration (oamg#1341)
- [IPU 7 -> 8] scangrubdevpartitionlayout: Skip warning msgs from fdisk (oamg#1306)
- [IPU 8 -> 9] Fix problems with the bootloader when upgrading to RHEL 9.6 on ARM (oamg#1275, oamg#1331, oamg#1339)
- [IPU 9 -> 10] Fix output of commands executed inside systemd-nspawn containers (oamg#1323)
- [IPU 9 -> 10] Fix remediation instructions for deprecated NM configuration (oamg#1330)
- [IPU 9 -> 10] Obsolete RHEL9 GPG key signed with SHA1 (oamg#1325)

### Enhancements
- Add RHEL 9.6 and 10.0 product certificates (oamg#1287, oamg#1309)
- Introduce new IPU paths 8.10 -> 9.6 and 9.6 -> 10.0 (oamg#1309)
- Add possibility to use net.naming-scheme (for IPU 8 -> 9 only for now) (oamg#1215, oamg#1312)
- Drop upgrade paths related to RHEL 8.8, 9.2, and 9.5 (oamg#1309, oamg#1344)
- Enable upgrade for EL8+ systems with LUKS bound to Clevis with TPM 2.0 token (oamg#1200)
- Enable upgrade with RHUI on Alibaba cloud for ARM machines (oamg#1277)
- Introduce a possibility to configure leapp actors covering RHUI on clouds (oamg#1142)
- Minor improvements in preupgrade reports (oamg#1315, oamg#1326, oamg#1342)
- Raise an inhibitor if unsupported target version supplied instead of error (oamg#1328)
- Skip checking of (PKI) `directory-hash` dir to speedup the upgrade process and clean logs (oamg#1297)
- Update leapp upgrade data files (oamg#1307)
- [IPU 9 -> 10] Cover upgrades for MySQL and PostgreSQL databases (oamg#1316, oamg#1343)
- [IPU 9 -> 10] Detect OpenSSL engines configured in /etc/pki/tls/openssl.cnf (oamg#1338)
- [IPU 9 -> 10] Detect XFS file systems with problematic parameters (oamg#1318)
- [IPU 9 -> 10] Detect deprecated network-scripts (ifcfg) files (oamg#1332, oamg#1347)
- [IPU 9 -> 10] Detect whether subscribed systems are using SCA (oamg#1333)
- [IPU 9 -> 10] Handle correctly switch of symlink to directory when upgrading Ruby IRB (oamg#1304)
- [IPU 9 -> 10] Inform user about Libdb removal if present (oamg#1319)
- [IPU 9 -> 10] Inhibit the upgrade for Intel CPUs with microarchitecture x86_64-v2 (oamg#1196)
- [IPU 9 -> 10] Move crypto-policies related actors to the system_upgrade_common repository to apply it for IPU 9 -> 10 as well
- [IPU 9 -> 10] Update checks for SAP HANA (oamg#1346)
- [IPU 9 -> 10] Update pam_userdb database backend for RHEL10 (oamg#1289)
pirat89 added a commit to pirat89/leapp-repository that referenced this pull request Feb 14, 2025
## Announce
This is the last upstream release covering IPU 7 -> 8. The
system_upgrade_el7toel8 repository is going to be removed from further
releases. Other artifacts related to IPU 7 -> 8 are going to be removed too.

In the same time it's the last release compatible with Python 2.7. Since now
we are going to develope and test the code only for Python 3.6 and newer.

## Packaging
- [IPU 9 -> 10] Require libdb-utils on RHEL 9 (oamg#1289)
- Require leapp-framework 6.0+ (oamg#1142)
- Update leapp-deps package to satisfy leapp_framework_deps 6 (oamg#1142)

## Upgrade handling
### Fixes
- Activate LVM VGs with `--sysinit` option to correct the use in the upgrade initramfs (oamg#1329)
- Cap max size of the sparse files to 1TiB when storage with large amount of space (oamg#1294)
- Fix IPU being blocked by resource limitations (oamg#1256)
- Fix pes events scanner crashing when there are duplicate packages in the received instructions (oamg#1296)
- Fix pes events scanner not respecting user’s transaction configuration (oamg#1296)
- Fix storage scanner crashing when command outputs contain colon character (oamg#1258)
- Fix the report when handling broken parsing of kernel cmdline (oamg#1290)
- Generate proper error message instead ModelViolationError when parsing invalid repository definition (oamg#1266)
- Handle default kernel cmdline when multiple boot entries for the default kernel are defined (oamg#1302)
- Load all available obsoleted keys (oamg#1286)
- Prevent a possible crash with LiveMode when adding the upgrade boot entry on LVM systems (oamg#1313)
- Prevent failure for upgrade paths when obsoleted keys are not defined (oamg#1285)
- [IPU 7 -> 8] Ignore invalid firewalld configuration (oamg#1341)
- [IPU 7 -> 8] scangrubdevpartitionlayout: Skip warning msgs from fdisk (oamg#1306)
- [IPU 8 -> 9] Fix problems with the bootloader when upgrading to RHEL 9.6 on ARM (oamg#1275, oamg#1331, oamg#1339)
- [IPU 9 -> 10] Fix output of commands executed inside systemd-nspawn containers (oamg#1323)
- [IPU 9 -> 10] Fix remediation instructions for deprecated NM configuration (oamg#1330)
- [IPU 9 -> 10] Obsolete RHEL9 GPG key signed with SHA1 (oamg#1325)

### Enhancements
- Add RHEL 9.6 and 10.0 product certificates (oamg#1287, oamg#1309)
- Introduce new IPU paths 8.10 -> 9.6 and 9.6 -> 10.0 (oamg#1309)
- Add possibility to use net.naming-scheme (for IPU 8 -> 9 only for now) (oamg#1215, oamg#1312)
- Drop upgrade paths related to RHEL 8.8, 9.2, and 9.5 (oamg#1309, oamg#1344)
- Enable upgrade for EL8+ systems with LUKS bound to Clevis with TPM 2.0 token (oamg#1200)
- Enable upgrade with RHUI on Alibaba cloud for ARM machines (oamg#1277)
- Introduce a possibility to configure leapp actors covering RHUI on clouds (oamg#1142)
- Minor improvements in preupgrade reports (oamg#1315, oamg#1326, oamg#1342)
- Raise an inhibitor if unsupported target version supplied instead of error (oamg#1328)
- Skip checking of (PKI) `directory-hash` dir to speedup the upgrade process and clean logs (oamg#1297)
- Update leapp upgrade data files (oamg#1307)
- [IPU 9 -> 10] Cover upgrades for MySQL and PostgreSQL databases (oamg#1316, oamg#1343)
- [IPU 9 -> 10] Detect OpenSSL engines configured in /etc/pki/tls/openssl.cnf (oamg#1338)
- [IPU 9 -> 10] Detect XFS file systems with problematic parameters (oamg#1318)
- [IPU 9 -> 10] Detect deprecated network-scripts (ifcfg) files (oamg#1332, oamg#1347)
- [IPU 9 -> 10] Detect whether subscribed systems are using SCA (oamg#1333)
- [IPU 9 -> 10] Handle correctly switch of symlink to directory when upgrading Ruby IRB (oamg#1304)
- [IPU 9 -> 10] Inform user about Libdb removal if present (oamg#1319)
- [IPU 9 -> 10] Inhibit the upgrade for Intel CPUs with microarchitecture x86_64-v2 (oamg#1196)
- [IPU 9 -> 10] Move crypto-policies related actors to the system_upgrade_common repository to apply it for IPU 9 -> 10 as well
- [IPU 9 -> 10] Update checks for SAP HANA (oamg#1346)
- [IPU 9 -> 10] Update pam_userdb database backend for RHEL10 (oamg#1289)

## Additional changes interesting for devels
- Introducing new upstream documentation focused on the leapp-repository
  project at https://leapp-repository.readthedocs.io/introduce-docs/.
  The documentation is still under reconstruction at this time, however we
  consider it a big step. All PRs impacting stuff that should be documented
  will be required to do so before they could be merged.
@pirat89 pirat89 mentioned this pull request Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants