Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: make nightly build pipeline faster #1958

Merged
merged 20 commits into from
Aug 21, 2024
Merged

chore: make nightly build pipeline faster #1958

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
4597734
Make target 'image' should work on both arm64 and amd64 architectures
BulkBeing Aug 19, 2024
cea2fb2
Use cargo-chef for caching in docker build
BulkBeing Aug 19, 2024
cd0b02b
Build rust binary in separately
BulkBeing Aug 20, 2024
aecf0a6
Build rust binary in separate job
BulkBeing Aug 20, 2024
746e691
Output logs for troubleshooting test failure
BulkBeing Aug 20, 2024
f8e9c3e
Disable pod deletion on test failure for debugging
BulkBeing Aug 20, 2024
07f257b
Make Rust binary executable
BulkBeing Aug 20, 2024
5483c18
Merge branch 'main' into rust-build-pipeline-2
vigith Aug 20, 2024
665823f
Merge branch 'main' into rust-build-pipeline-2
BulkBeing Aug 21, 2024
ae83f54
Merge branch 'main' into rust-build-pipeline-2
BulkBeing Aug 21, 2024
a92f0ca
Revert changes made for debugging
BulkBeing Aug 21, 2024
cced395
Build Rust binaries outside buildkit in nightly build pipelines
BulkBeing Aug 21, 2024
baa9f07
Build Rust binaries outside buildkit in release pipelines
BulkBeing Aug 21, 2024
6919c99
Right way to get arch.
whynowy Aug 21, 2024
74c4cf5
Merge branch 'main' into rust-build-pipeline-2
BulkBeing Aug 21, 2024
68a5f38
Pin to ubuntu-22.04 for Rust builds, Remove debug statements from Mak…
BulkBeing Aug 21, 2024
77d587c
chore: pass linker as arg
vigith Aug 21, 2024
1cafc74
chore: linker as arg
vigith Aug 21, 2024
671d4ea
Specify toolchain in rust-toolchain.toml
BulkBeing Aug 21, 2024
724c222
New line at the end in rust-toolchain.toml is needed for pipelines to…
BulkBeing Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,42 @@ jobs:
- run: make lint
- run: git diff --exit-code

build-rust-amd64:
runs-on: ubuntu-22.04
defaults:
run:
working-directory: ./rust
steps:
- uses: actions/checkout@v4
- name: Setup Rust toolchain
uses: actions-rust-lang/[email protected]
with:
rustflags: ''
- name: Configure sccache
run: |
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
- name: Run sccache-cache
uses: mozilla-actions/[email protected]
- name: Install dependencies
run: sudo apt-get install -y protobuf-compiler
- name: Build binary
run: RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu
- name: Rename binary
run: cp -pv target/x86_64-unknown-linux-gnu/release/numaflow ./numaflow-rs-linux-amd64
- name: List files
run: pwd && ls -al && file ./numaflow-rs-linux-amd64
- name: Upload numaflow binary
uses: actions/upload-artifact@v4
with:
name: numaflow-rs-linux-amd64
path: rust/numaflow-rs-linux-amd64
if-no-files-found: error

e2e-tests:
name: E2E Tests
runs-on: ubuntu-latest
needs: [ build-rust-amd64 ]
timeout-minutes: 20
strategy:
fail-fast: false
Expand Down Expand Up @@ -185,6 +218,10 @@ jobs:
with:
path: ui/node_modules
key: ${{ runner.os }}-node-dep-v1-${{ hashFiles('**/yarn.lock') }}
- name: Download Rust amd64 binaries
uses: actions/download-artifact@v4
with:
name: numaflow-rs-linux-amd64
- name: Install k3d
run: curl -sfL https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash &
- name: Create a cluster
Expand Down
79 changes: 76 additions & 3 deletions .github/workflows/nightly-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ defaults:
shell: bash

jobs:
build-binaries:
build-go-binaries:
runs-on: ubuntu-20.04
if: github.repository == 'numaproj/numaflow'
name: Build binaries
Expand All @@ -40,9 +40,70 @@ jobs:
name: binaries
path: dist

build-rust-amd64:
runs-on: ubuntu-22.04
defaults:
run:
working-directory: ./rust
steps:
- uses: actions/checkout@v4
- name: Setup Rust toolchain
uses: actions-rust-lang/[email protected]
with:
rustflags: ''
- name: Configure sccache
run: |
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
- name: Run sccache-cache
uses: mozilla-actions/[email protected]
- name: Install dependencies
run: sudo apt-get install -y protobuf-compiler
- name: Build binary
run: RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu
- name: Rename binary
run: cp -pv target/x86_64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-amd64
- name: Upload numaflow binary
uses: actions/upload-artifact@v3
with:
name: numaflow-rs-linux-amd64
path: rust/numaflow-rs-linux-amd64

build-rust-arm64:
runs-on: ubuntu-22.04
defaults:
run:
working-directory: ./rust
steps:
- uses: actions/checkout@v4
- name: Update Rust Toolchain Target
run: |
echo "targets = ['aarch64-unknown-linux-gnu']" >> rust-toolchain.toml
- name: Setup Rust toolchain
uses: actions-rust-lang/[email protected]
with:
rustflags: ''
- name: Configure sccache
run: |
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
- name: Run sccache-cache
uses: mozilla-actions/[email protected]
- name: Install dependenices
run: sudo apt-get install -y gcc-aarch64-linux-gnu protobuf-compiler
- name: Build binary
run: RUSTFLAGS='-C target-feature=+crt-static -C linker=aarch64-linux-gnu-gcc' cargo build --release --target aarch64-unknown-linux-gnu
- name: Rename binary
run: cp -pv target/aarch64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-arm64
- name: Upload numaflow binary
uses: actions/upload-artifact@v3
with:
name: numaflow-rs-linux-arm64
path: rust/numaflow-rs-linux-arm64

build-push-linux-multi:
name: Build & push linux/amd64 and linux/arm64
needs: [ build-binaries ]
needs: [ build-go-binaries, build-rust-amd64, build-rust-arm64]
runs-on: ubuntu-20.04
if: github.repository == 'numaproj/numaflow'
strategy:
Expand All @@ -63,12 +124,24 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Download binaries
- name: Download Go binaries
uses: actions/download-artifact@v3
with:
name: binaries
path: dist/

- name: Download Rust amd64 binaries
uses: actions/download-artifact@v3
with:
name: numaflow-rs-linux-amd64
path: dist/numaflow-rs-linux-amd64

- name: Download Rust arm64 binaries
uses: actions/download-artifact@v3
with:
name: numaflow-rs-linux-arm64
path: dist/numaflow-rs-linux-arm64

- name: Registry Login
uses: docker/login-action@v2
with:
Expand Down
67 changes: 64 additions & 3 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ defaults:
shell: bash

jobs:
build-binaries:
build-go-binaries:
runs-on: ubuntu-20.04
if: github.repository == 'numaproj/numaflow'
name: Build binaries
Expand All @@ -38,9 +38,58 @@ jobs:
name: binaries
path: dist

build-rust-amd64:
runs-on: ubuntu-22.04
defaults:
run:
working-directory: ./rust
steps:
- uses: actions/checkout@v4
- name: Setup Rust toolchain
uses: actions-rust-lang/[email protected]
with:
rustflags: ''
- name: Install dependencies
run: sudo apt-get install -y protobuf-compiler
- name: Build binary
run: RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu
- name: Rename binary
run: cp -pv target/x86_64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-amd64
- name: Upload numaflow binary
uses: actions/upload-artifact@v3
with:
name: numaflow-rs-linux-amd64
path: rust/numaflow-rs-linux-amd64

build-rust-arm64:
runs-on: ubuntu-22.04
defaults:
run:
working-directory: ./rust
steps:
- uses: actions/checkout@v4
- name: Update Rust Toolchain Target
run: |
echo "targets = ['aarch64-unknown-linux-gnu']" >> rust-toolchain.toml
- name: Setup Rust toolchain
uses: actions-rust-lang/[email protected]
with:
rustflags: ''
- name: Install dependenices
run: sudo apt-get install -y gcc-aarch64-linux-gnu protobuf-compiler
- name: Build binary
run: RUSTFLAGS='-C target-feature=+crt-static -C linker=aarch64-linux-gnu-gcc' cargo build --release --target aarch64-unknown-linux-gnu
- name: Rename binary
run: cp -pv target/aarch64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-arm64
- name: Upload numaflow binary
uses: actions/upload-artifact@v3
with:
name: numaflow-rs-linux-arm64
path: rust/numaflow-rs-linux-arm64

build-push-linux-multi:
name: Build & push linux/amd64 and linux/arm64
needs: [ build-binaries ]
needs: [ build-go-binaries, build-rust-amd64, build-rust-arm64]
runs-on: ubuntu-20.04
if: github.repository == 'numaproj/numaflow'
strategy:
Expand All @@ -61,12 +110,24 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Download binaries
- name: Download Go binaries
uses: actions/download-artifact@v3
with:
name: binaries
path: dist/

- name: Download Rust amd64 binaries
uses: actions/download-artifact@v3
with:
name: numaflow-rs-linux-amd64
path: dist/numaflow-rs-linux-amd64

- name: Download Rust arm64 binaries
uses: actions/download-artifact@v3
with:
name: numaflow-rs-linux-arm64
path: dist/numaflow-rs-linux-arm64

- name: Registry Login
uses: docker/login-action@v2
with:
Expand Down
51 changes: 16 additions & 35 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,41 +8,30 @@ ARG ARCH
RUN apk update && apk upgrade && \
apk add ca-certificates && \
apk --no-cache add tzdata
ARG ARCH

COPY dist/numaflow-linux-${ARCH} /bin/numaflow
COPY dist/numaflow-rs-linux-${ARCH} /bin/numaflow-rs

RUN chmod +x /bin/numaflow
RUN chmod +x /bin/numaflow-rs

####################################################################################################
# extension base
# Rust binary
####################################################################################################
FROM rust:1.80-bookworm AS extension-base
FROM lukemathwalker/cargo-chef:latest-rust-1.80 AS chef
ARG TARGETPLATFORM

RUN apt-get update && apt-get install protobuf-compiler -y

RUN cargo new numaflow
# Create a new empty shell project
WORKDIR /numaflow
RUN apt-get update && apt-get install -y protobuf-compiler

RUN cargo new servesink
COPY ./rust/servesink/Cargo.toml ./servesink/

RUN cargo new backoff
COPY ./rust/backoff/Cargo.toml ./backoff/

RUN cargo new numaflow-models
COPY ./rust/numaflow-models/Cargo.toml ./numaflow-models/

RUN cargo new monovertex
COPY ./rust/monovertex/Cargo.toml ./monovertex/
FROM chef AS planner
COPY ./rust/ .
RUN cargo chef prepare --recipe-path recipe.json

RUN cargo new serving
COPY ./rust/serving/Cargo.toml ./serving/Cargo.toml

# Copy all Cargo.toml and Cargo.lock files for caching dependencies
COPY ./rust/Cargo.toml ./rust/Cargo.lock ./
FROM chef AS rust-builder
ARG TARGETPLATFORM
ARG ARCH
COPY --from=planner /numaflow/recipe.json recipe.json

# Build to cache dependencies
RUN --mount=type=cache,target=/usr/local/cargo/registry \
Expand All @@ -52,18 +41,10 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
"linux/arm64") TARGET="aarch64-unknown-linux-gnu" ;; \
*) echo "Unsupported platform: ${TARGETPLATFORM}" && exit 1 ;; \
esac && \
mkdir -p src/bin && echo "fn main() {}" > src/bin/main.rs && \
RUSTFLAGS='-C target-feature=+crt-static' cargo build --workspace --all --release --target ${TARGET}
RUSTFLAGS='-C target-feature=+crt-static' cargo chef cook --workspace --release --target ${TARGET} --recipe-path recipe.json

# Copy the actual source code files of the main project and the subprojects
COPY ./rust/src ./src
COPY ./rust/servesink/src ./servesink/src
COPY ./rust/backoff/src ./backoff/src
COPY ./rust/numaflow-models/src ./numaflow-models/src
COPY ./rust/serving/src ./serving/src
COPY ./rust/monovertex/src ./monovertex/src
COPY ./rust/monovertex/build.rs ./monovertex/build.rs
COPY ./rust/monovertex/proto ./monovertex/proto
COPY ./rust/ .

# Build the real binaries
RUN --mount=type=cache,target=/usr/local/cargo/registry \
Expand All @@ -73,7 +54,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
"linux/arm64") TARGET="aarch64-unknown-linux-gnu" ;; \
*) echo "Unsupported platform: ${TARGETPLATFORM}" && exit 1 ;; \
esac && \
touch src/bin/main.rs && \
RUSTFLAGS='-C target-feature=+crt-static' cargo build --workspace --all --release --target ${TARGET} && \
cp -pv target/${TARGET}/release/numaflow /root/numaflow

Expand All @@ -82,13 +62,14 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
####################################################################################################
ARG BASE_IMAGE
FROM ${BASE_IMAGE} AS numaflow
ARG ARCH

COPY --from=base /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=base /bin/numaflow /bin/numaflow
COPY --from=base /bin/numaflow-rs /bin/numaflow-rs
COPY ui/build /ui/build

COPY --from=extension-base /root/numaflow /bin/numaflow-rs
COPY ./rust/serving/config config

ENTRYPOINT [ "/bin/numaflow" ]
Expand Down
Loading
Loading