-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stage beam_Publish_Python_SDK_Distroless_Snapshots.yml #33470
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
92 changes: 92 additions & 0 deletions
92
.github/workflows/beam_Publish_Python_SDK_Distroless_Snapshots.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
name: Publish Beam Python SDK Distroless Snapshots | ||
|
||
on: | ||
schedule: | ||
- cron: '45 */8 * * *' | ||
workflow_dispatch: | ||
|
||
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event | ||
permissions: | ||
actions: write | ||
pull-requests: read | ||
checks: read | ||
contents: read | ||
deployments: read | ||
id-token: none | ||
issues: read | ||
discussions: read | ||
packages: read | ||
pages: read | ||
repository-projects: read | ||
security-events: read | ||
statuses: read | ||
|
||
# This allows a subsequently queued workflow run to interrupt previous runs | ||
concurrency: | ||
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}' | ||
cancel-in-progress: true | ||
|
||
env: | ||
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }} | ||
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }} | ||
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }} | ||
docker_registry: gcr.io | ||
|
||
jobs: | ||
Python_SDK_Distroless_Snapshots: | ||
if: | | ||
github.event_name == 'workflow_dispatch' || | ||
(github.event_name == 'schedule' && github.repository == 'apache/beam') | ||
runs-on: [self-hosted, ubuntu-20.04, main] | ||
timeout-minutes: 160 | ||
name: ${{ matrix.job_name }} (${{ matrix.python_version }}) | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
job_name: ["Python_SDK_Distroless_Snapshots"] | ||
job_phrase: ["N/A"] | ||
python_version: | ||
- "python3.9" | ||
- "python3.10" | ||
- "python3.11" | ||
- "python3.12" | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Setup repository | ||
uses: ./.github/actions/setup-action | ||
with: | ||
comment_phrase: ${{ matrix.job_phrase }} | ||
github_token: ${{ secrets.GITHUB_TOKEN }} | ||
github_job: ${{ matrix.job_name }} (${{ matrix.python_version }}) | ||
- name: Find Beam Version | ||
# We extract the Beam version here and tag the containers with it. Version will be in the form "2.xx.y.dev". | ||
# This is needed to run pipelines that use the default environment at HEAD, for example, when a | ||
# pipeline uses an expansion service built from HEAD. | ||
run: | | ||
BEAM_VERSION_LINE=$(cat gradle.properties | grep "sdk_version") | ||
echo "BEAM_VERSION=${BEAM_VERSION_LINE#*sdk_version=}" >> $GITHUB_ENV | ||
- name: Set up Docker Buildx | ||
uses: docker/setup-buildx-action@v1 | ||
- name: GCloud Docker credential helper | ||
run: | | ||
gcloud auth configure-docker ${{ env.docker_registry }} | ||
# TODO(https://github.com/apache/beam/issues/32914): create after merging into main branch | ||
# - name: Build and push Python distroless image | ||
|
||
|
||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than creating a whole new workflow, can we just add these images to
beam/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
Line 72 in 637a42e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@damccorm it's better to keep this separate from the existing Beam Publish SDK snapshot workflow due to how its matrix is set up. Can we just move forward with this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I recommend we use the different workflow here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One huge workflow could be hard to run / debug.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand why this is easier to run/debug. If you add this to the matrix, it will get run as its own job and we won't need to maintain 2 workflows that do the same thing. I don't think we should move forward with this change and put up #33481 to revert; lets not merge PRs while there is active disagreement about the way to move forward.
We don't have to merge the revert, but could you please help me understand what the concerns are?
Could you help me understand why? I don't see anything incompatible right now, but if we need to adjust the structure of this workflow a little that is also ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
flaky is not what I care here. I think using one big workflow to cover all snapshots is basically bad.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok - why is it bad though? In general, these do the exact same thing, so I think having a single workflow makes a lot of sense. My view:
Pros of a single large workflow:
Cons:
Failure Rate = (individual failure rate)^(# jobs)
- I think this is fine (maybe even good) for a job that we want to have a flake rate approaching 0if
conditionsAre there pros/cons that I'm missing or do you value these things differently than me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what are we going to do if we have more ML package-related distroless images? or what if we start supporting other base images? Do we keep growing this workflow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we would keep growing this workflow - I still don't understand why this is bad, could you help me understand what your concerns with having many jobs in the workflow are?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll note that https://github.com/apache/beam/pull/33484/files#r1901809077 is relevant. If we have different workflows it encourages divergent paths like this, which I think are the wrong approach. If there is a good reason for not encapsulating this logic in a gradle task then I'd be happy to have a separate workflow.