Stage beam_Publish_Python_SDK_Distroless_Snapshots.yml

[damondouglas]

This PR addresses #32914 staging a GitHub workflow that aims to publish to a staging registry at gcr.io/apache-beam-testing as with the non-distroless variants. Currently nothing happens with this workflow as it needs to be merged before testing with self-hosted runners.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label build.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

Rather than creating a whole new workflow, can we just add these images to

beam/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml

Line 72 in 637a42e

- "python:container:py312"

? I don't think distroless should be any different here, and that will automatically parallelize everything (so there shouldn't be any sort of performance hit)

[damondouglas]

@damccorm it's better to keep this separate from the existing Beam Publish SDK snapshot workflow due to how its matrix is set up. Can we just move forward with this?

[liferoad]

I recommend we use the different workflow here.

[liferoad]

One huge workflow could be hard to run / debug.

[damccorm]

One huge workflow could be hard to run / debug.

I don't understand why this is easier to run/debug. If you add this to the matrix, it will get run as its own job and we won't need to maintain 2 workflows that do the same thing. I don't think we should move forward with this change and put up #33481 to revert; lets not merge PRs while there is active disagreement about the way to move forward.

We don't have to merge the revert, but could you please help me understand what the concerns are?

it's better to keep this separate from the existing Beam Publish SDK snapshot workflow due to how its matrix is set up. Can we just move forward with this?

Could you help me understand why? I don't see anything incompatible right now, but if we need to adjust the structure of this workflow a little that is also ok

[liferoad]

flaky is not what I care here. I think using one big workflow to cover all snapshots is basically bad.

[damccorm]

Ok - why is it bad though? In general, these do the exact same thing, so I think having a single workflow makes a lot of sense. My view:

Pros of a single large workflow:

• 1 thing to monitor
• Snapshots get published together at the same time, guaranteeing they will be taken for the same commit (this is actually pretty important IMO for a good x-lang story and making things easier to reason about)
• Shared logic exists in a single place instead of being copied in many workflow definitions (could be mitigated by sharing logic across workflows)

Cons:

• Flakes are more likely to compound Failure Rate = (individual failure rate)^(# jobs) - I think this is fine (maybe even good) for a job that we want to have a flake rate approaching 0
• It is harder to iterate if you just want a single language launched - this is rare, and also we can make this happen in a single workflow definition via if conditions

Are there pros/cons that I'm missing or do you value these things differently than me.

[liferoad]

what are we going to do if we have more ML package-related distroless images? or what if we start supporting other base images? Do we keep growing this workflow?

[damccorm]

Yes, we would keep growing this workflow - I still don't understand why this is bad, could you help me understand what your concerns with having many jobs in the workflow are?

[damccorm]

I'll note that https://github.com/apache/beam/pull/33484/files#r1901809077 is relevant. If we have different workflows it encourages divergent paths like this, which I think are the wrong approach. If there is a good reason for not encapsulating this logic in a gradle task then I'd be happy to have a separate workflow.