Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stage beam_Publish_Python_SDK_Distroless_Snapshots.yml #33470

Merged
merged 2 commits into from
Jan 2, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions .github/workflows/beam_Publish_Python_SDK_Distroless_Snapshots.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Publish Beam Python SDK Distroless Snapshots

on:
schedule:
- cron: '45 */8 * * *'
workflow_dispatch:

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: read
checks: read
contents: read
deployments: read
id-token: none
issues: read
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}'
cancel-in-progress: true

env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
docker_registry: gcr.io

jobs:
Python_SDK_Distroless_Snapshots:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than creating a whole new workflow, can we just add these images to

? I don't think distroless should be any different here, and that will automatically parallelize everything (so there shouldn't be any sort of performance hit)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@damccorm it's better to keep this separate from the existing Beam Publish SDK snapshot workflow due to how its matrix is set up. Can we just move forward with this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend we use the different workflow here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One huge workflow could be hard to run / debug.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One huge workflow could be hard to run / debug.

I don't understand why this is easier to run/debug. If you add this to the matrix, it will get run as its own job and we won't need to maintain 2 workflows that do the same thing. I don't think we should move forward with this change and put up #33481 to revert; lets not merge PRs while there is active disagreement about the way to move forward.

We don't have to merge the revert, but could you please help me understand what the concerns are?

it's better to keep this separate from the existing Beam Publish SDK snapshot workflow due to how its matrix is set up. Can we just move forward with this?

Could you help me understand why? I don't see anything incompatible right now, but if we need to adjust the structure of this workflow a little that is also ok

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flaky is not what I care here. I think using one big workflow to cover all snapshots is basically bad.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok - why is it bad though? In general, these do the exact same thing, so I think having a single workflow makes a lot of sense. My view:

Pros of a single large workflow:

  • 1 thing to monitor
  • Snapshots get published together at the same time, guaranteeing they will be taken for the same commit (this is actually pretty important IMO for a good x-lang story and making things easier to reason about)
  • Shared logic exists in a single place instead of being copied in many workflow definitions (could be mitigated by sharing logic across workflows)

Cons:

  • Flakes are more likely to compound Failure Rate = (individual failure rate)^(# jobs) - I think this is fine (maybe even good) for a job that we want to have a flake rate approaching 0
  • It is harder to iterate if you just want a single language launched - this is rare, and also we can make this happen in a single workflow definition via if conditions

Are there pros/cons that I'm missing or do you value these things differently than me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are we going to do if we have more ML package-related distroless images? or what if we start supporting other base images? Do we keep growing this workflow?

Copy link
Contributor

@damccorm damccorm Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we would keep growing this workflow - I still don't understand why this is bad, could you help me understand what your concerns with having many jobs in the workflow are?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll note that https://github.com/apache/beam/pull/33484/files#r1901809077 is relevant. If we have different workflows it encourages divergent paths like this, which I think are the wrong approach. If there is a good reason for not encapsulating this logic in a gradle task then I'd be happy to have a separate workflow.

if: |
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'schedule' && github.repository == 'apache/beam')
runs-on: [self-hosted, ubuntu-20.04, main]
timeout-minutes: 160
name: ${{ matrix.job_name }} (${{ matrix.python_version }})
strategy:
fail-fast: false
matrix:
job_name: ["Python_SDK_Distroless_Snapshots"]
job_phrase: ["N/A"]
python_version:
- "python3.9"
- "python3.10"
- "python3.11"
- "python3.12"
steps:
- uses: actions/checkout@v4
- name: Setup repository
uses: ./.github/actions/setup-action
with:
comment_phrase: ${{ matrix.job_phrase }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ matrix.job_name }} (${{ matrix.python_version }})
- name: Find Beam Version
# We extract the Beam version here and tag the containers with it. Version will be in the form "2.xx.y.dev".
# This is needed to run pipelines that use the default environment at HEAD, for example, when a
# pipeline uses an expansion service built from HEAD.
run: |
BEAM_VERSION_LINE=$(cat gradle.properties | grep "sdk_version")
echo "BEAM_VERSION=${BEAM_VERSION_LINE#*sdk_version=}" >> $GITHUB_ENV
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: GCloud Docker credential helper
run: |
gcloud auth configure-docker ${{ env.docker_registry }}
# TODO(https://github.com/apache/beam/issues/32914): create after merging into main branch
# - name: Build and push Python distroless image



Loading