Skip to content

Commit

Permalink
MDBF-866 - Create bintar builder from non-EOL builder
Browse files Browse the repository at this point in the history
AlmaLinux 8 was chosen as the oldest non-eol distro for a new bintar which will build += 11.7
Buildbot configuration changes will follow after this PR is merged. (new builder,worker).

A dedicated workflow was defined, with a workflow_call directive to allow us
to rebuild the image anytime we like.

About bintar.Dockerfile
- it is based on the centos.Dockerfile with small changes to remove redundant code
- use galera-4 from RHEL-8 because we no longer build on alma/rocky, neither
the server or the galera library. Compatibility wise, should be ok, because
we also test RHEL packages on alma/rocky VM's.

About static libs:
- static libraries are build in a separate stage to reduce the image size
- the general rule is to install the -devel packages (provide headers) then
to compile the static library and move it in a path used then by CMAKE
- the strategy to compile static is to take the source rpm and then install deps + re-build.
Different libs have different needs in terms of what custom
modifications of the SPEC files are needed.

Testing:
- one can build the image from scratch, compile the server and check resulted binaries with LDD
- the list of *.sh files introduced by this patch is the minimum I found that will produce a binary package similar
with what we've released so far.
  • Loading branch information
RazvanLiviuVarzaru committed Jan 21, 2025
1 parent d7792d9 commit eb56650
Show file tree
Hide file tree
Showing 9 changed files with 209 additions and 0 deletions.
43 changes: 43 additions & 0 deletions .github/workflows/build-bintar.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: Build Bintar image

on:
push:
branches:
- 'main'
- 'dev'
paths:
- 'ci_build_images/bintar.Dockerfile'
- 'ci_build_images/qpress.Dockerfile'
- 'ci_build_images/buildbot-worker.Dockerfile'
- 'ci_build_images/scripts/**'
- .github/workflows/build-bintar.yml
- .github/workflows/bbw_build_container_template.yml
pull_request:
paths:
- 'ci_build_images/bintar.Dockerfile'
- 'ci_build_images/qpress.Dockerfile'
- 'ci_build_images/buildbot-worker.Dockerfile'
- 'ci_build_images/scripts/**'
- .github/workflows/build-bintar.yml
- .github/workflows/bbw_build_container_template.yml
workflow_call:

jobs:
build-images:
strategy:
fail-fast: false
matrix:
include:
- image: almalinux:8
platforms: linux/amd64
tag: almalinux8-bintar
nogalera: false

uses: ./.github/workflows/bbw_build_container_template.yml
with:
dockerfile: bintar.Dockerfile
image: ${{ matrix.image }}
platforms: ${{ matrix.platforms }}
tag: ${{ matrix.tag }}
nogalera: ${{ matrix.nogalera }}
secrets: inherit
90 changes: 90 additions & 0 deletions ci_build_images/bintar.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Buildbot worker for building MariaDB
#
# Provides a bintar image based on AlmaLinux with build dependencies
# and statically compiled libraries

ARG BASE_IMAGE
FROM "$BASE_IMAGE" AS buildeps
LABEL maintainer="MariaDB Buildbot maintainers"

# Install updates and required packages
# hadolint ignore=SC2086
RUN dnf -y install 'dnf-command(config-manager)' \
&& source /etc/os-release \
&& ARCH=amd64 \
&& dnf -y --enablerepo=extras install epel-release \
&& dnf config-manager --set-enabled powertools \
&& dnf -y module enable mariadb-devel \
&& dnf -y install almalinux-release-devel \
&& VERSION_ID=-${VERSION_ID} \
&& VERSION_ID=${VERSION_ID%%.*} \
&& dnf config-manager --add-repo https://ci.mariadb.org/galera/mariadb-4.x-latest-gal-${ARCH}-rhel${VERSION_ID}.repo \
&& dnf -y upgrade \
&& dnf -y groupinstall "Development Tools" \
&& dnf -y builddep mariadb-server \
&& dnf -y install \
asio-devel \
buildbot-worker \
bzip2 \
bzip2-devel \
ccache \
check-devel \
cracklib-devel \
createrepo \
curl-devel \
eigen3-devel \
flex \
galera-4 \
gnutls-devel \
java-1.8.0-openjdk-devel \
java-1.8.0-openjdk \
jemalloc-devel \
libcurl-devel \
libevent-devel \
libffi-devel \
libpmem-devel \
libxml2-devel \
libzstd-devel \
perl-autodie \
perl-Net-SSLeay \
python3-devel \
readline-devel \
rpmlint \
ruby \
snappy-devel \
subversion \
unixODBC \
unixODBC-devel \
wget \
which \
xz-devel \
yum-utils \
&& dnf clean all \
&& curl -sL "https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_$(uname -m)" >/usr/local/bin/dumb-init \
&& chmod +x /usr/local/bin/dumb-init
ENV WSREP_PROVIDER=/usr/lib64/galera-4/libgalera_smm.so

## Build the static libraries in a separate stage so we save space
FROM buildeps AS staticlibs

COPY ./scripts/* /scripts/
WORKDIR /scripts
RUN chmod +x ./*.sh

# Build static libraries
RUN mkdir -p ./local/lib/ \
&& ./libaio.sh \
&& ./liblz4.sh \
&& ./xz.sh \
&& ./ncurses.sh \
&& ./libpmem.sh \
&& ./libzstd.sh \
&& ./gnutls.sh

FROM buildeps AS bintar
COPY --from=staticlibs /scripts/local/lib /scripts/local/lib

### Other .Dockerfiles will be concatenated here (worker, qpress, etc)



13 changes: 13 additions & 0 deletions ci_build_images/scripts/gnutls.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash

set -e

yumdownloader --source gnutls-devel
yum-builddep -y gnutls-*.src.rpm
rpm -ivh gnutls-*.src.rpm
sed -i 's/--disable-static/--enable-static \\\n--disable-tests/g' ~/rpmbuild/SPECS/gnutls.spec
sed -i 's/export LDFLAGS="-Wl,--no-add-needed"/export LDFLAGS="-Wl,--copy-dt-needed-entries -fpic -fPIC"/g' ~/rpmbuild/SPECS/gnutls.spec
sed -i 's/make %{?_smp_mflags}/make %{?_smp_mflags} CFLAGS="-fpic -fPIC" CXX_FLAGS="-fpic -fPIC"/g' ~/rpmbuild/SPECS/gnutls.spec
rpmbuild -bc ~/rpmbuild/SPECS/gnutls.spec
mv -v ~/rpmbuild/BUILD/gnutls-*/lib/.libs/libgnutls.a local/lib
rm -rf ~/rpmbuild gnutls-*.src.rpm
9 changes: 9 additions & 0 deletions ci_build_images/scripts/libaio.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

set -e

yumdownloader --source libaio-devel
yum-builddep -y libaio-*.src.rpm
rpmbuild --recompile libaio-*.src.rpm
mv -v ~/rpmbuild/BUILD/libaio-*/src/*.a local/lib
rm -rf ~/rpmbuild libaio-*.src.rpm
11 changes: 11 additions & 0 deletions ci_build_images/scripts/liblz4.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

set -e

yumdownloader --source lz4-devel
yum-builddep -y lz4-*.src.rpm
rpm -ivh lz4-*.src.rpm
sed -i 's/%make_build/%make_build CFLAGS="-fpic -fPIC" CXX_FLAGS="-fpic -fPIC"/g' ~/rpmbuild/SPECS/lz4.spec
rpmbuild -bc ~/rpmbuild/SPECS/lz4.spec
mv -v ~/rpmbuild/BUILD/lz4-*/lib/liblz4.a local/lib
rm -rf ~/rpmbuild lz4-*.src.rpm
9 changes: 9 additions & 0 deletions ci_build_images/scripts/libpmem.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

set -e

yumdownloader --source libpmem-devel
yum-builddep -y pmdk-*.src.rpm
rpmbuild --recompile pmdk-*.src.rpm
mv -v ~/rpmbuild/BUILD/pmdk-*/src/nondebug/libpmem.a local/lib
rm -rf ~/rpmbuild pmdk-*.src.rpm
11 changes: 11 additions & 0 deletions ci_build_images/scripts/libzstd.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/bash

set -e

yumdownloader --source libzstd
yum-builddep -y zstd-*.src.rpm
rpm -ivh zstd-*.src.rpm
rpmbuild -bp ~/rpmbuild/SPECS/zstd.spec
make CFLAGS=-fPIC -C ~/rpmbuild/BUILD/zstd-*
mv -v ~/rpmbuild/BUILD/zstd-*/lib/libzstd.a local/lib
rm -rf ~/rpmbuild zstd-*.src.rpm
9 changes: 9 additions & 0 deletions ci_build_images/scripts/ncurses.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

set -e

yumdownloader --source ncurses-devel
yum-builddep -y ncurses-*.src.rpm
rpmbuild --recompile ncurses-*.src.rpm
mv -v ~/rpmbuild/BUILDROOT/ncurses-*/usr/lib64/*.a local/lib
rm -rf ~/rpmbuild ncurses-*.src.rpm
14 changes: 14 additions & 0 deletions ci_build_images/scripts/xz.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash

set -e

yumdownloader --source xz-devel
yum-builddep -y xz-*.src.rpm
rpm -ivh xz-*.src.rpm
sed -i 's/--disable-static/--disable-shared --enable-static --with-pic/g' ~/rpmbuild/SPECS/xz.spec
sed -i 's/CFLAGS="/CFLAGS="-fpic -fPIC /g' ~/rpmbuild/SPECS/xz.spec
sed -i 's/export CFLAGS/export CFLAGS\nCXXFLAGS="-fpic -fPIC"\nexport CXXFLAGS/g' ~/rpmbuild/SPECS/xz.spec
sed -i '/\*\.a/d' ~/rpmbuild/SPECS/xz.spec
rpmbuild -bc ~/rpmbuild/SPECS/xz.spec
mv -v ~/rpmbuild/BUILD/xz-*/src/liblzma/.libs/liblzma.a local/lib
rm -rf ~/rpmbuild xz-*.src.rpm

0 comments on commit eb56650

Please sign in to comment.