Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support creating mashes from multiple tags #10

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Support creating mashes from multiple tags #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ecdb82a
Include support for multiple snapshot repo
clefrks Oct 9, 2019
4b06ff5
Remove hardcoded dist- value
clefrks Oct 9, 2019
fa8efd4
Add $MASH_DIR_OLD and $MASH_TRACKER_DIR folder
clefrks Oct 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions koji-setup/bootstrap-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,19 @@ if [[ -n "$SRC_RPM_DIR" && -n "$BIN_RPM_DIR" ]]; then
find "$DEBUG_RPM_DIR" -name "*.$RPM_ARCH.rpm" | xargs -n 1 -I {} sudo -u kojiadmin koji import {}
fi
fi
sudo -u kojiadmin koji add-tag dist-"$TAG_NAME"
sudo -u kojiadmin koji edit-tag dist-"$TAG_NAME" -x mock.package_manager=dnf
sudo -u kojiadmin koji add-tag "$TAG_NAME"
sudo -u kojiadmin koji edit-tag "$TAG_NAME" -x mock.package_manager=dnf
if [[ -n "$SRC_RPM_DIR" && -n "$BIN_RPM_DIR" ]]; then
sudo -u kojiadmin koji list-pkgs --quiet | xargs -I {} sudo -u kojiadmin koji add-pkg --owner kojiadmin dist-"$TAG_NAME" {}
sudo -u kojiadmin koji list-untagged | xargs -n 1 -I {} sudo -u kojiadmin koji call tagBuildBypass dist-"$TAG_NAME" {}
sudo -u kojiadmin koji list-pkgs --quiet | xargs -I {} sudo -u kojiadmin koji add-pkg --owner kojiadmin "$TAG_NAME" {}
sudo -u kojiadmin koji list-untagged | xargs -n 1 -I {} sudo -u kojiadmin koji call tagBuildBypass "$TAG_NAME" {}
fi
sudo -u kojiadmin koji add-tag --parent dist-"$TAG_NAME" --arches "$RPM_ARCH" dist-"$TAG_NAME"-build
sudo -u kojiadmin koji add-target dist-"$TAG_NAME" dist-"$TAG_NAME"-build
sudo -u kojiadmin koji add-group dist-"$TAG_NAME"-build build
sudo -u kojiadmin koji add-group dist-"$TAG_NAME"-build srpm-build
sudo -u kojiadmin koji add-group-pkg dist-"$TAG_NAME"-build build autoconf automake automake-dev binutils bzip2 clr-rpm-config coreutils diffutils gawk gcc gcc-dev gettext gettext-bin git glibc-dev glibc-locale glibc-utils grep gzip hostname libc6-dev libcap libtool libtool-dev linux-libc-headers m4 make netbase nss-altfiles patch pigz pkg-config pkg-config-dev rpm-build sed sed-doc shadow systemd-lib tar unzip which xz
sudo -u kojiadmin koji add-group-pkg dist-"$TAG_NAME"-build srpm-build coreutils cpio curl-bin git glibc-utils grep gzip make pigz plzip rpm-build sed shadow tar unzip wget xz
sudo -u kojiadmin koji add-tag --parent "$TAG_NAME" --arches "$RPM_ARCH" "$TAG_NAME"-build
sudo -u kojiadmin koji add-target "$TAG_NAME" "$TAG_NAME"-build
sudo -u kojiadmin koji add-group "$TAG_NAME"-build build
sudo -u kojiadmin koji add-group "$TAG_NAME"-build srpm-build
sudo -u kojiadmin koji add-group-pkg "$TAG_NAME"-build build autoconf automake automake-dev binutils bzip2 clr-rpm-config coreutils diffutils gawk gcc gcc-dev gettext gettext-bin git glibc-dev glibc-locale glibc-utils grep gzip hostname libc6-dev libcap libtool libtool-dev linux-libc-headers m4 make netbase nss-altfiles patch pigz pkg-config pkg-config-dev rpm-build sed sed-doc shadow systemd-lib tar unzip which xz
sudo -u kojiadmin koji add-group-pkg "$TAG_NAME"-build srpm-build coreutils cpio curl-bin git glibc-utils grep gzip make pigz plzip rpm-build sed shadow tar unzip wget xz
if [[ -n "$EXTERNAL_REPO" ]]; then
sudo -u kojiadmin koji add-external-repo -t dist-"$TAG_NAME"-build dist-"$TAG_NAME"-external-repo "$EXTERNAL_REPO"
sudo -u kojiadmin koji add-external-repo -t "$TAG_NAME"-build "$TAG_NAME"-external-repo "$EXTERNAL_REPO"
fi
sudo -u kojiadmin koji regen-repo dist-"$TAG_NAME"-build
sudo -u kojiadmin koji regen-repo "$TAG_NAME"-build
10 changes: 5 additions & 5 deletions koji-setup/deploy-mash.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,15 @@ repodir = file://$KOJI_DIR
use_sqlite = True
use_repoview = False
EOF
cat > /etc/mash/clear.mash <<- EOF
cat > /etc/mash/$TAG_NAME.mash <<- EOF
[clear]
rpm_path = %(arch)s/os/Packages
repodata_path = %(arch)s/os/
source_path = source/SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = dist-$TAG_NAME
tag = $TAG_NAME
inherit = True
strict_keys = False
arches = $RPM_ARCH
Expand All @@ -46,19 +46,19 @@ EOF
mkdir -p "$MASH_SCRIPT_DIR"
cp -f "$SCRIPT_DIR"/mash.sh "$MASH_SCRIPT_DIR"
mkdir -p /etc/systemd/system
cat > /etc/systemd/system/mash.service <<- EOF
cat > /etc/systemd/system/mash@$TAG_NAME.service <<- EOF
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for a case like this, we want the service file on disk to be named [email protected] to indicate to systemd that we want to pass a parameter, but not hard-code the parameter for all instantiations of the service.

[Unit]
Description=Mash script to loop local repository creation for local image builds

[Service]
User=kojiadmin
Group=kojiadmin
ExecStart=$MASH_SCRIPT_DIR/mash.sh
ExecStart=${MASH_SCRIPT_DIR}/mash.sh %I
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now mash
systemctl enable --now mash@$TAG_NAME
31 changes: 17 additions & 14 deletions koji-setup/mash.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@
set -e
. /etc/profile.d/proxy.sh || :

KOJI_TAG="${KOJI_TAG:-"$1"}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

"${1:?"Missing tag parameter"}"

Is probably a more accurate parameter expansion here. If the service is enabled without a parameter, we want to indicate that this is required and have the script fail.

BUILD_ARCH="${BUILD_ARCH:-x86_64}"
KOJI_DIR="${KOJI_DIR:-/srv/koji}"
MASH_DIR="${MASH_DIR:-/srv/mash}"
MASH_TRACKER_FILE="$MASH_DIR"/latest-mash-build
MASH_TRACKER_FILE="$MASH_DIR"/latest-${KOJI_TAG}-build
Comment on lines 11 to +12
Copy link
Contributor

@gtkramer gtkramer Oct 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I am understanding the intent of the PR correctly, I think we want

MASH_DIR="${MASH_DIR:-"/srv/mash/$KOJI_TAG"}"

Then we wouldn't modify MASH_TRACKER_FILE, but rather, self-contain a similar structure to every mash in the name of a folder that varies by tag. This allows for more fine-grained management of each repo and allows multiple repos for each tag to build up independently, if so desired.

MASH_TRACKER_DIR="$MASH_DIR"/latest
MASH_DIR_OLD="$MASH_TRACKER_DIR".old
MASH_DIR_NEW="$MASH_TRACKER_DIR".new
Expand Down Expand Up @@ -37,18 +38,18 @@ create_dist_repos() {

cp -f "${KOJI_REPO_PATH}/groups/comps.xml" "${comps_file}"

make_repo "${source_dir}" "${output_dir}" "clear/${BUILD_ARCH}/os" "Packages" "${bin_rpm_paths}" "${comps_file}" &
make_repo "${source_dir}" "${output_dir}" "clear/${BUILD_ARCH}/debug" "." "${debuginfo_rpm_paths}" &
make_repo "${source_dir}" "${output_dir}" "clear/source/SRPMS" "." "${src_rpm_paths}" &
make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/${BUILD_ARCH}/os" "Packages" "${bin_rpm_paths}" "${comps_file}" &
make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/${BUILD_ARCH}/debug" "." "${debuginfo_rpm_paths}" &
make_repo "${source_dir}" "${output_dir}" "${KOJI_TAG}/source/SRPMS" "." "${src_rpm_paths}" &
wait

create_dnf_conf "${work_dir}/dnf-os.conf" "${output_dir}/clear/${BUILD_ARCH}/os" clear-os
create_dnf_conf "${work_dir}/dnf-debug.conf" "${output_dir}/clear/${BUILD_ARCH}/debug" clear-debug
create_dnf_conf "${work_dir}/dnf-SRPMS.conf" "${output_dir}/clear/source/SRPMS" clear-SRPMS
create_dnf_conf "${work_dir}/dnf-os.conf" "${output_dir}/${KOJI_TAG}/${BUILD_ARCH}/os" clear-os
create_dnf_conf "${work_dir}/dnf-debug.conf" "${output_dir}/${KOJI_TAG}/${BUILD_ARCH}/debug" clear-debug
create_dnf_conf "${work_dir}/dnf-SRPMS.conf" "${output_dir}/${KOJI_TAG}/source/SRPMS" clear-SRPMS

write_packages_file "${work_dir}/dnf-os.conf" "$output_dir/clear/$BUILD_ARCH/packages-os"
write_packages_file "${work_dir}/dnf-debug.conf" "$output_dir/clear/$BUILD_ARCH/packages-debug"
write_packages_file "${work_dir}/dnf-SRPMS.conf" "$output_dir/clear/source/packages-SRPMS"
write_packages_file "${work_dir}/dnf-os.conf" "$output_dir/${KOJI_TAG}/$BUILD_ARCH/packages-os"
write_packages_file "${work_dir}/dnf-debug.conf" "$output_dir/${KOJI_TAG}/$BUILD_ARCH/packages-debug"
write_packages_file "${work_dir}/dnf-SRPMS.conf" "$output_dir/${KOJI_TAG}/source/packages-SRPMS"
Comment on lines -40 to +52
Copy link
Contributor

@gtkramer gtkramer Oct 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these function calls, we unfortunately need to keep the clear/ structure. For better or worse, a lot of our processes today depend on this naming convention. I don't think the change to these lines would be needed if we contain the mash structure in a root folder name that varies by tag.


rm -rf "${work_dir}"
}
Expand Down Expand Up @@ -99,18 +100,20 @@ if [[ -e "$MASH_TRACKER_FILE" ]]; then
else
MASH_BUILD_NUM=0
fi
KOJI_TAG="${KOJI_TAG:-"dist-clear"}"
KOJI_REPO_PATH="$(realpath "$KOJI_DIR/repos/$KOJI_TAG-build/latest")"
KOJI_BUILD_NUM="$(basename "$KOJI_REPO_PATH")"
if [[ "$MASH_BUILD_NUM" -ne "$KOJI_BUILD_NUM" ]]; then
rm -rf "$MASH_DIR_NEW"
mkdir -p "$MASH_DIR_NEW"
create_dist_repos "$MASH_TRACKER_DIR" "$MASH_DIR_NEW"
if [[ -e "$MASH_TRACKER_DIR" ]]; then
mv "$MASH_TRACKER_DIR" "$MASH_DIR_OLD"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since $MASH_DIR_OLD needs to exist now as the move is moving it to a subdirectory rather than just renaming the directory to .old I think this needs a mkdir -p $MASH_DIR_OLD.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes! i will update that and $MASH_TRACKER_DIR!
thanks!

if [[ -e "$MASH_TRACKER_DIR/$KOJI_TAG" ]]; then
mkdir -p "$MASH_DIR_OLD"
mv "$MASH_TRACKER_DIR/$KOJI_TAG" "$MASH_DIR_OLD/$KOJI_TAG"
fi
mv "$MASH_DIR_NEW" "$MASH_TRACKER_DIR"
mkdir -p "$MASH_TRACKER_DIR"
mv "$MASH_DIR_NEW/$KOJI_TAG" "$MASH_TRACKER_DIR"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly with $MASH_TRACKER_DIR being used as a subdir, the script will need to ensure it is created first.

rm -rf "$MASH_DIR_OLD"
rm -rf "$MASH_DIR_NEW"
Comment on lines -109 to +116
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with the change to the root folder, none of these changes are needed?


echo "$KOJI_BUILD_NUM" > "$MASH_TRACKER_FILE"
fi