-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: make nightly build pipeline faster #1958
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sreekanth <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1958 +/- ##
==========================================
- Coverage 60.66% 53.64% -7.03%
==========================================
Files 317 286 -31
Lines 28159 22319 -5840
==========================================
- Hits 17084 11974 -5110
+ Misses 10137 9400 -737
- Partials 938 945 +7 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
whynowy
reviewed
Aug 21, 2024
Signed-off-by: Derek Wang <[email protected]>
…efile Signed-off-by: Sreekanth <[email protected]>
vigith
reviewed
Aug 21, 2024
Signed-off-by: Vigith Maurice <[email protected]>
vigith
reviewed
Aug 21, 2024
vigith
approved these changes
Aug 21, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
awesome 🎉
Signed-off-by: Vigith Maurice <[email protected]>
Signed-off-by: Sreekanth <[email protected]>
… work Signed-off-by: Sreekanth <[email protected]>
whynowy
approved these changes
Aug 21, 2024
whynowy
added a commit
that referenced
this pull request
Sep 3, 2024
Signed-off-by: Sreekanth <[email protected]> Signed-off-by: Derek Wang <[email protected]> Signed-off-by: Vigith Maurice <[email protected]> Co-authored-by: Vigith Maurice <[email protected]> Co-authored-by: Derek Wang <[email protected]>
SaniyaKalamkar
pushed a commit
to SaniyaKalamkar/numaflow
that referenced
this pull request
Jan 19, 2025
Signed-off-by: Sreekanth <[email protected]> Signed-off-by: Derek Wang <[email protected]> Signed-off-by: Vigith Maurice <[email protected]> Co-authored-by: Vigith Maurice <[email protected]> Co-authored-by: Derek Wang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #1943
Same as #1949 . Trying smaller changes at a time, since I'm not able to figure out why tests for monovertex is failing in CI #1949
UPDATE: The CI was failing becuase the Rust binary within the image didn't had executable permission.
Summary
x86_64
architecture. This results in very slow build forarm64
binary within buildkit's emulation.dist/
directory whennumaflow
container image is built (same as the pattern we follow for Go binary).Dockerfile
if we add/remove a cargo workspace.release
pipeline.image
target inMakefile
is modified to work in bothamd64
andarm64
hosts.These changes should reduce the nightly builds time from current average of 1hr 30 minutes to around 5 minutes.
In my testing of the nightly build pipeline, a fresh build completes in under 7 minutes. Subsequent builds (with high sccache hits) finishes in under 4 minutes.
Current nightly builds:
![Screenshot 2024-08-21 at 10 23 02 AM](https://private-user-images.githubusercontent.com/32409051/359754863-a9c1a085-ee72-473a-96a1-7cfd106a4332.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkwMzkwODIsIm5iZiI6MTczOTAzODc4MiwicGF0aCI6Ii8zMjQwOTA1MS8zNTk3NTQ4NjMtYTljMWEwODUtZWU3Mi00NzNhLTk2YTEtN2NmZDEwNmE0MzMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA4VDE4MTk0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY3MGVjY2E0ZDM4MGJkNjE1NjI4YTBmYTQzNzFkN2ZlMzg1OTM3ODFlYjJlNWQxZTI1YWJkMDcyZTNhZmJjNDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.sEjBByyeo2Cfe2GVkvKPLlLG9YPObLygB6rxlP_OIcw)
Fresh nightly build with the changes:
![Screenshot 2024-08-21 at 10 36 12 AM](https://private-user-images.githubusercontent.com/32409051/359754884-31aafee6-6625-4b6f-90c0-6465d7834858.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkwMzkwODIsIm5iZiI6MTczOTAzODc4MiwicGF0aCI6Ii8zMjQwOTA1MS8zNTk3NTQ4ODQtMzFhYWZlZTYtNjYyNS00YjZmLTkwYzAtNjQ2NWQ3ODM0ODU4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA4VDE4MTk0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWRkMTk5MGUzNzZjMWViOTBmYjRhYzk0OGIzNjdhYjkzOTJiZjM2MWVjNThiN2IwMTA4OGMwZGIzYzZmZDM0ZDEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.pCunhAOu7xFKiFp_7uSDt5b0z6VRIlyQiO3MWkJy3T8)
Nightly build with high
![Screenshot 2024-08-21 at 10 36 33 AM](https://private-user-images.githubusercontent.com/32409051/359754909-a6912948-f47d-4d09-8a19-7d7741f88fa1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkwMzkwODIsIm5iZiI6MTczOTAzODc4MiwicGF0aCI6Ii8zMjQwOTA1MS8zNTk3NTQ5MDktYTY5MTI5NDgtZjQ3ZC00ZDA5LThhMTktN2Q3NzQxZjg4ZmExLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA4VDE4MTk0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRiODRiNmMxODlmNTI3YjM4OTNkNmE0ZmRlOGE3ZTY4OTdmNmQzNDM2ZWIyZjU4ZjAyMjMwY2ZhNGY1NDg0OGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.k_d3DRzeFtfz1rrTxCQ-gLvw1j7MQbnfC5RY-lP9HyY)
sccache
hit rate: