Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround for Boost compilation error with Clang 18 and libc++ in ossfuzz buildpack-deps #15713

Merged
merged 3 commits into from
Jan 21, 2025
Merged

Workaround for Boost compilation error with Clang 18 and libc++ in ossfuzz buildpack-deps #15713

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2993728
Replace base-clang by ossfuzz base-builder
r0qs Jan 13, 2025
429e994
Bump Z3 and CVC5
r0qs Jan 17, 2025
837208c
Convert spaces to tabs
r0qs Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 63 additions & 74 deletions scripts/docker/buildpack-deps/Dockerfile.ubuntu.clang.ossfuzz
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,10 @@
#
# (c) 2016-2021 solidity contributors.
#------------------------------------------------------------------------------
FROM gcr.io/oss-fuzz-base/base-clang:latest as base
LABEL version="8"
# FIXME: Workaround for Boost 1.83.0 build failure when using Clang 18 (default since https://github.com/google/oss-fuzz/pull/11714)
# See: https://github.com/google/oss-fuzz/blob/5a9322260c4c66e7eb3e5d922b9fee6621ffd8da/projects/solidity/Dockerfile#L17C1-L18C139
FROM gcr.io/oss-fuzz-base/base-builder@sha256:19782f7fe8092843368894dbc471ce9b30dd6a2813946071a36e8b05f5b1e27e AS base
r0qs marked this conversation as resolved.
Show resolved Hide resolved
LABEL version="9"

ARG DEBIAN_FRONTEND=noninteractive

Expand All @@ -44,6 +46,7 @@ RUN apt-get update; \
openjdk-8-jdk \
pkg-config \
python3-pip \
python3-dev \
software-properties-common \
sudo \
texinfo \
Expand All @@ -59,50 +62,39 @@ RUN apt-get update; \
pygments-lexer-solidity \
pylint \
requests \
tabulate \
z3-solver;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Z3 is already installed from source in the steps below, so this is unnecessary. I also removed CMake, as it is already included in the base image.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the python package of Z3, which is unrelated to the Z3 binary built below.
However, if it can be removed without anything breaking, that it's good to remove it :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it is not used by any CI job based on the ossfuzz image as far as I know. The chk_proofs which is the job that runs the script mentioned here: #15551 (comment) uses the Ubuntu 24.04 image and it is not run by ossfuzz. This is why I'm also wondering if all the smt stuff is really needed here, or if we should just remove it from this image.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I updated Z3 and CVC5, but I kept the removal of the z3-python, I don't think it is relevant for this image. Because it is required by the tests under test/formal, which indeed uses the z3 python bindings, but I believe the fuzzer does not run such tests. Maybe there are more things that we could cleanup as well.


# Install cmake 3.21.2 (minimum requirement is cmake 3.10)
RUN wget https://github.com/Kitware/CMake/releases/download/v3.21.2/cmake-3.21.2-Linux-x86_64.sh; \
test "$(sha256sum cmake-3.21.2-Linux-x86_64.sh)" = "3310362c6fe4d4b2dc00823835f3d4a7171bbd73deb7d059738494761f1c908c cmake-3.21.2-Linux-x86_64.sh"; \
chmod +x cmake-3.21.2-Linux-x86_64.sh; \
./cmake-3.21.2-Linux-x86_64.sh --skip-license --prefix="/usr"
tabulate;

FROM base AS libraries

# Boost
RUN set -ex; \
cd /usr/src; \
wget -q 'https://archives.boost.io/release/1.74.0/source/boost_1_74_0.tar.bz2' -O boost.tar.bz2; \
test "$(sha256sum boost.tar.bz2)" = "83bfc1507731a0906e387fc28b7ef5417d591429e51e788417fe9ff025e116b1 boost.tar.bz2" && \
tar -xf boost.tar.bz2; \
rm boost.tar.bz2; \
cd boost_1_74_0; \
CXXFLAGS="-stdlib=libc++ -pthread" LDFLAGS="-stdlib=libc++" ./bootstrap.sh --with-toolset=clang --prefix=/usr; \
./b2 toolset=clang cxxflags="-stdlib=libc++ -pthread" linkflags="-stdlib=libc++ -pthread" headers; \
./b2 toolset=clang cxxflags="-stdlib=libc++ -pthread" linkflags="-stdlib=libc++ -pthread" \
link=static variant=release runtime-link=static \
system filesystem unit_test_framework program_options \
install -j $(($(nproc)/2)); \
rm -rf /usr/src/boost_1_74_0
cd /usr/src; \
wget -q 'https://archives.boost.io/release/1.83.0/source/boost_1_83_0.tar.bz2' -O boost.tar.bz2; \
test "$(sha256sum boost.tar.bz2)" = "6478edfe2f3305127cffe8caf73ea0176c53769f4bf1585be237eb30798c3b8e boost.tar.bz2" && \
tar -xf boost.tar.bz2; \
rm boost.tar.bz2; \
cd boost_1_83_0; \
CXXFLAGS="-std=c++17 -stdlib=libc++ -pthread" LDFLAGS="-stdlib=libc++" ./bootstrap.sh --with-toolset=clang --prefix=/usr; \
./b2 toolset=clang cxxflags="-stdlib=libc++ -pthread" linkflags="-stdlib=libc++ -pthread" headers; \
./b2 toolset=clang cxxflags="-stdlib=libc++ -pthread" linkflags="-stdlib=libc++ -pthread" \
link=static variant=release runtime-link=static \
system filesystem unit_test_framework program_options \
install -j $(($(nproc)/2)); \
rm -rf /usr/src/boost_1_83_0

# Z3
RUN set -ex; \
git clone --depth 1 -b z3-4.12.1 https://github.com/Z3Prover/z3.git \
/usr/src/z3; \
cd /usr/src/z3; \
mkdir build; \
cd build; \
LDFLAGS=$CXXFLAGS cmake -DZ3_BUILD_LIBZ3_SHARED=OFF -DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_BUILD_TYPE=Release ..; \
make libz3 -j; \
make install; \
rm -rf /usr/src/z3
z3_version="4.13.3"; \
z3_archive_name="z3-${z3_version}-x64-glibc-2.35"; \
wget "https://github.com/Z3Prover/z3/releases/download/z3-${z3_version}/${z3_archive_name}.zip" -O /opt/z3.zip; \
test "$(sha256sum /opt/z3.zip)" = "32c7377026733c9d7b33c21cd77a68f50ba682367207b031a6bfd80140a8722f /opt/z3.zip"; \
unzip -j /opt/z3.zip "${z3_archive_name}/bin/z3" -d /usr/bin; \
rm -f /opt/z3.zip;

# Eldarica
RUN set -ex; \
apt-get update; \
apt-get install -qy unzip openjdk-11-jre; \
apt-get install -qqy openjdk-11-jre; \
eldarica_version="2.1"; \
wget "https://github.com/uuverifiers/eldarica/releases/download/v${eldarica_version}/eldarica-bin-${eldarica_version}.zip" -O /opt/eld_binaries.zip; \
test "$(sha256sum /opt/eld_binaries.zip)" = "0ac43f45c0925383c9d2077f62bbb515fd792375f3b2b101b30c9e81dcd7785c /opt/eld_binaries.zip"; \
Expand All @@ -111,26 +103,23 @@ RUN set -ex; \

# CVC5
RUN set -ex; \
cvc5_version="1.1.2"; \
wget "https://github.com/cvc5/cvc5/releases/download/cvc5-${cvc5_version}/cvc5-Linux-static.zip" -O /opt/cvc5.zip; \
test "$(sha256sum /opt/cvc5.zip)" = "cf291aef67da8eaa8d425a51f67f3f72f36db8b1040655dc799b64e3d69e6086 /opt/cvc5.zip"; \
unzip /opt/cvc5.zip -x "cvc5-Linux-static/lib/cmake/*" -d /opt; \
mv /opt/cvc5-Linux-static/bin/* /usr/bin; \
mv /opt/cvc5-Linux-static/include/* /usr/include; \
mv /opt/cvc5-Linux-static/lib/* /usr/lib; \
rm -rf /opt/cvc5-Linux-static /opt/cvc5.zip;
cvc5_version="1.2.0"; \
cvc5_archive_name="cvc5-Linux-x86_64-static"; \
wget "https://github.com/cvc5/cvc5/releases/download/cvc5-${cvc5_version}/${cvc5_archive_name}.zip" -O /opt/cvc5.zip; \
test "$(sha256sum /opt/cvc5.zip)" = "d18f174ff9a11923c32c3f871f844ed16bd77a28f51050b8e7c8d821c98a1c2e /opt/cvc5.zip"; \
unzip -j /opt/cvc5.zip "${cvc5_archive_name}/bin/cvc5" -d /usr/bin; \
rm -f /opt/cvc5.zip;

# OSSFUZZ: libprotobuf-mutator
RUN set -ex; \
git clone https://github.com/google/libprotobuf-mutator.git \
/usr/src/libprotobuf-mutator; \
git clone https://github.com/google/libprotobuf-mutator.git /usr/src/libprotobuf-mutator; \
cd /usr/src/libprotobuf-mutator; \
git checkout 3521f47a2828da9ace403e4ecc4aece1a84feb36; \
mkdir build; \
cd build; \
cmake .. -GNinja -DLIB_PROTO_MUTATOR_DOWNLOAD_PROTOBUF=ON \
-DLIB_PROTO_MUTATOR_TESTING=OFF -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="/usr"; \
-DLIB_PROTO_MUTATOR_TESTING=OFF -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="/usr"; \
ninja; \
cp -vpr external.protobuf/bin/* /usr/bin/; \
cp -vpr external.protobuf/include/* /usr/include/; \
Expand All @@ -152,37 +141,37 @@ RUN set -ex; \

# gmp
RUN set -ex; \
# Replace system installed libgmp static library
# with sanitized version. Do not perform apt
# remove because it removes mlton as well that
# we need for building libabicoder
cd /usr/src/; \
git clone --depth 1 --branch gmp-6.2.1 https://github.com/gmp-mirror/gmp-6.2 gmp/; \
rm -r gmp/.git/; \
test \
"$(tar --create gmp/ --sort=name --mtime=1970-01-01Z --owner=0 --group=0 --numeric-owner | sha256sum)" = \
"d606ff6a4ce98692f9920031e85ea8fcf4a65ce1426f6f0048b8794aefed174b -"; \
# NOTE: This removes also libgmp.so, which git depends on
rm -f /usr/lib/x86_64-linux-gnu/libgmp.*; \
rm -f /usr/include/x86_64-linux-gnu/gmp.h; \
cd gmp/; \
autoreconf -i; \
./configure --prefix=/usr --enable-static=yes --enable-maintainer-mode; \
make -j; \
make check; \
make install; \
rm -rf /usr/src/gmp/
# Replace system installed libgmp static library
# with sanitized version. Do not perform apt
# remove because it removes mlton as well that
# we need for building libabicoder
cd /usr/src/; \
git clone --depth 1 --branch gmp-6.2.1 https://github.com/gmp-mirror/gmp-6.2 gmp/; \
rm -r gmp/.git/; \
test \
"$(tar --create gmp/ --sort=name --mtime=1970-01-01Z --owner=0 --group=0 --numeric-owner | sha256sum)" = \
"d606ff6a4ce98692f9920031e85ea8fcf4a65ce1426f6f0048b8794aefed174b -"; \
# NOTE: This removes also libgmp.so, which git depends on
rm -f /usr/lib/x86_64-linux-gnu/libgmp.*; \
rm -f /usr/include/x86_64-linux-gnu/gmp.h; \
cd gmp/; \
autoreconf -i; \
./configure --prefix=/usr --enable-static=yes --enable-maintainer-mode; \
make -j; \
make check; \
make install; \
rm -rf /usr/src/gmp/

# libabicoder
RUN set -ex; \
cd /usr/src; \
git clone https://github.com/ekpyron/Yul-Isabelle; \
cd Yul-Isabelle; \
cd libabicoder; \
CXX=clang++ CXXFLAGS="-stdlib=libc++ -pthread" make; \
cp libabicoder.a /usr/lib; \
cp abicoder.hpp /usr/include; \
rm -rf /usr/src/Yul-Isabelle
cd /usr/src; \
git clone https://github.com/ekpyron/Yul-Isabelle; \
cd Yul-Isabelle; \
cd libabicoder; \
CXX=clang++ CXXFLAGS="-stdlib=libc++ -pthread" make; \
cp libabicoder.a /usr/lib; \
cp abicoder.hpp /usr/include; \
rm -rf /usr/src/Yul-Isabelle

FROM base
COPY --from=libraries /usr/lib /usr/lib
Expand Down
Loading