-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MDBF-866 - Create bintar builder from non-EOL builder
AlmaLinux 8 was chosen as the oldest non-eol distro for a new bintar which will build += 11.7 Buildbot configuration changes will follow after this PR is merged. (new builder,worker). A dedicated workflow was defined, with a workflow_call directive to allow us to rebuild the image anytime we like. About bintar.Dockerfile - it is based on the centos.Dockerfile with small changes to remove redundant code - use galera-4 from RHEL-8 because we no longer build on alma/rocky, neither the server or the galera library. Compatibility wise, should be ok, because we also test RHEL packages on alma/rocky VM's. About static libs: - static libraries are build in a separate stage to reduce the image size - the general rule is to install the -devel packages (provide headers) then to compile the static library and move it in a path used then by CMAKE - the strategy to compile static is to take the source rpm and then install deps + re-build. Different libs have different needs in terms of what custom modifications of the SPEC files are needed. Testing: - one can build the image from scratch, compile the server and check resulted binaries with LDD - the list of *.sh files introduced by this patch is the minimum I found that will produce a binary package similar with what we've released so far.
- Loading branch information
1 parent
d7792d9
commit eb56650
Showing
9 changed files
with
209 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
name: Build Bintar image | ||
|
||
on: | ||
push: | ||
branches: | ||
- 'main' | ||
- 'dev' | ||
paths: | ||
- 'ci_build_images/bintar.Dockerfile' | ||
- 'ci_build_images/qpress.Dockerfile' | ||
- 'ci_build_images/buildbot-worker.Dockerfile' | ||
- 'ci_build_images/scripts/**' | ||
- .github/workflows/build-bintar.yml | ||
- .github/workflows/bbw_build_container_template.yml | ||
pull_request: | ||
paths: | ||
- 'ci_build_images/bintar.Dockerfile' | ||
- 'ci_build_images/qpress.Dockerfile' | ||
- 'ci_build_images/buildbot-worker.Dockerfile' | ||
- 'ci_build_images/scripts/**' | ||
- .github/workflows/build-bintar.yml | ||
- .github/workflows/bbw_build_container_template.yml | ||
workflow_call: | ||
|
||
jobs: | ||
build-images: | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
include: | ||
- image: almalinux:8 | ||
platforms: linux/amd64 | ||
tag: almalinux8-bintar | ||
nogalera: false | ||
|
||
uses: ./.github/workflows/bbw_build_container_template.yml | ||
with: | ||
dockerfile: bintar.Dockerfile | ||
image: ${{ matrix.image }} | ||
platforms: ${{ matrix.platforms }} | ||
tag: ${{ matrix.tag }} | ||
nogalera: ${{ matrix.nogalera }} | ||
secrets: inherit |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,90 @@ | ||
# Buildbot worker for building MariaDB | ||
# | ||
# Provides a bintar image based on AlmaLinux with build dependencies | ||
# and statically compiled libraries | ||
|
||
ARG BASE_IMAGE | ||
FROM "$BASE_IMAGE" AS buildeps | ||
LABEL maintainer="MariaDB Buildbot maintainers" | ||
|
||
# Install updates and required packages | ||
# hadolint ignore=SC2086 | ||
RUN dnf -y install 'dnf-command(config-manager)' \ | ||
&& source /etc/os-release \ | ||
&& ARCH=amd64 \ | ||
&& dnf -y --enablerepo=extras install epel-release \ | ||
&& dnf config-manager --set-enabled powertools \ | ||
&& dnf -y module enable mariadb-devel \ | ||
&& dnf -y install almalinux-release-devel \ | ||
&& VERSION_ID=-${VERSION_ID} \ | ||
&& VERSION_ID=${VERSION_ID%%.*} \ | ||
&& dnf config-manager --add-repo https://ci.mariadb.org/galera/mariadb-4.x-latest-gal-${ARCH}-rhel${VERSION_ID}.repo \ | ||
&& dnf -y upgrade \ | ||
&& dnf -y groupinstall "Development Tools" \ | ||
&& dnf -y builddep mariadb-server \ | ||
&& dnf -y install \ | ||
asio-devel \ | ||
buildbot-worker \ | ||
bzip2 \ | ||
bzip2-devel \ | ||
ccache \ | ||
check-devel \ | ||
cracklib-devel \ | ||
createrepo \ | ||
curl-devel \ | ||
eigen3-devel \ | ||
flex \ | ||
galera-4 \ | ||
gnutls-devel \ | ||
java-1.8.0-openjdk-devel \ | ||
java-1.8.0-openjdk \ | ||
jemalloc-devel \ | ||
libcurl-devel \ | ||
libevent-devel \ | ||
libffi-devel \ | ||
libpmem-devel \ | ||
libxml2-devel \ | ||
libzstd-devel \ | ||
perl-autodie \ | ||
perl-Net-SSLeay \ | ||
python3-devel \ | ||
readline-devel \ | ||
rpmlint \ | ||
ruby \ | ||
snappy-devel \ | ||
subversion \ | ||
unixODBC \ | ||
unixODBC-devel \ | ||
wget \ | ||
which \ | ||
xz-devel \ | ||
yum-utils \ | ||
&& dnf clean all \ | ||
&& curl -sL "https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_$(uname -m)" >/usr/local/bin/dumb-init \ | ||
&& chmod +x /usr/local/bin/dumb-init | ||
ENV WSREP_PROVIDER=/usr/lib64/galera-4/libgalera_smm.so | ||
|
||
## Build the static libraries in a separate stage so we save space | ||
FROM buildeps AS staticlibs | ||
|
||
COPY ./scripts/* /scripts/ | ||
WORKDIR /scripts | ||
RUN chmod +x ./*.sh | ||
|
||
# Build static libraries | ||
RUN mkdir -p ./local/lib/ \ | ||
&& ./libaio.sh \ | ||
&& ./liblz4.sh \ | ||
&& ./xz.sh \ | ||
&& ./ncurses.sh \ | ||
&& ./libpmem.sh \ | ||
&& ./libzstd.sh \ | ||
&& ./gnutls.sh | ||
|
||
FROM buildeps AS bintar | ||
COPY --from=staticlibs /scripts/local/lib /scripts/local/lib | ||
|
||
### Other .Dockerfiles will be concatenated here (worker, qpress, etc) | ||
|
||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source gnutls-devel | ||
yum-builddep -y gnutls-*.src.rpm | ||
rpm -ivh gnutls-*.src.rpm | ||
sed -i 's/--disable-static/--enable-static \\\n--disable-tests/g' ~/rpmbuild/SPECS/gnutls.spec | ||
sed -i 's/export LDFLAGS="-Wl,--no-add-needed"/export LDFLAGS="-Wl,--copy-dt-needed-entries -fpic -fPIC"/g' ~/rpmbuild/SPECS/gnutls.spec | ||
sed -i 's/make %{?_smp_mflags}/make %{?_smp_mflags} CFLAGS="-fpic -fPIC" CXX_FLAGS="-fpic -fPIC"/g' ~/rpmbuild/SPECS/gnutls.spec | ||
rpmbuild -bc ~/rpmbuild/SPECS/gnutls.spec | ||
mv -v ~/rpmbuild/BUILD/gnutls-*/lib/.libs/libgnutls.a local/lib | ||
rm -rf ~/rpmbuild gnutls-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source libaio-devel | ||
yum-builddep -y libaio-*.src.rpm | ||
rpmbuild --recompile libaio-*.src.rpm | ||
mv -v ~/rpmbuild/BUILD/libaio-*/src/*.a local/lib | ||
rm -rf ~/rpmbuild libaio-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source lz4-devel | ||
yum-builddep -y lz4-*.src.rpm | ||
rpm -ivh lz4-*.src.rpm | ||
sed -i 's/%make_build/%make_build CFLAGS="-fpic -fPIC" CXX_FLAGS="-fpic -fPIC"/g' ~/rpmbuild/SPECS/lz4.spec | ||
rpmbuild -bc ~/rpmbuild/SPECS/lz4.spec | ||
mv -v ~/rpmbuild/BUILD/lz4-*/lib/liblz4.a local/lib | ||
rm -rf ~/rpmbuild lz4-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source libpmem-devel | ||
yum-builddep -y pmdk-*.src.rpm | ||
rpmbuild --recompile pmdk-*.src.rpm | ||
mv -v ~/rpmbuild/BUILD/pmdk-*/src/nondebug/libpmem.a local/lib | ||
rm -rf ~/rpmbuild pmdk-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source libzstd | ||
yum-builddep -y zstd-*.src.rpm | ||
rpm -ivh zstd-*.src.rpm | ||
rpmbuild -bp ~/rpmbuild/SPECS/zstd.spec | ||
make CFLAGS=-fPIC -C ~/rpmbuild/BUILD/zstd-* | ||
mv -v ~/rpmbuild/BUILD/zstd-*/lib/libzstd.a local/lib | ||
rm -rf ~/rpmbuild zstd-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source ncurses-devel | ||
yum-builddep -y ncurses-*.src.rpm | ||
rpmbuild --recompile ncurses-*.src.rpm | ||
mv -v ~/rpmbuild/BUILDROOT/ncurses-*/usr/lib64/*.a local/lib | ||
rm -rf ~/rpmbuild ncurses-*.src.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
|
||
yumdownloader --source xz-devel | ||
yum-builddep -y xz-*.src.rpm | ||
rpm -ivh xz-*.src.rpm | ||
sed -i 's/--disable-static/--disable-shared --enable-static --with-pic/g' ~/rpmbuild/SPECS/xz.spec | ||
sed -i 's/CFLAGS="/CFLAGS="-fpic -fPIC /g' ~/rpmbuild/SPECS/xz.spec | ||
sed -i 's/export CFLAGS/export CFLAGS\nCXXFLAGS="-fpic -fPIC"\nexport CXXFLAGS/g' ~/rpmbuild/SPECS/xz.spec | ||
sed -i '/\*\.a/d' ~/rpmbuild/SPECS/xz.spec | ||
rpmbuild -bc ~/rpmbuild/SPECS/xz.spec | ||
mv -v ~/rpmbuild/BUILD/xz-*/src/liblzma/.libs/liblzma.a local/lib | ||
rm -rf ~/rpmbuild xz-*.src.rpm |