Improve nightly build and deployment logic #3164

zaneselvans · 2023-12-15T18:55:12Z

Overview

Remove minor versions from the GHA we depend on to minimize dependabot churn.
Remove the logic that was updating conda lockfiles on push.
Purge old files from distribution paths before uploading new outputs. This prevents stale files from hanging around after we change the name of a file.
Tag commits that are used for nightly builds.
Update stable branch to point at newly tagged release
Update nightly branch to point at newly successful nightly build tag.
Simplify Dockerfile / make it more internally consistent

Issues / Questions:

The nightly build jank continues. We need to dedicate some resources to consolidating it!

Part of #3140

jdangerx

Stoked for this to get in, I think probably making a limited bot token makes the most sense for auth!

.github/workflows/build-deploy-pudl.yml

zaneselvans

I've got something here in all the places I think we'd need to update to get the build and release tagging working:

vYYYY.MM.DD tags are created manually.
nightly-YYYY-MM-DD tags are created by the workflow, before the build is kicked off.
The nightly branch (which is currently a tag) has to be updated from within the Docker container, but we don't currently have a git repo inside the Docker container.
The stable branch (also currently a tag) will be updated by the release workflow.

Questions in review comments.

.github/workflows/pytest.yml

.github/workflows/build-deploy-pudl.yml

.github/workflows/release.yml

.pre-commit-config.yaml

.github/workflows/update-conda-lockfile.yml

docker/gcp_pudl_etl.sh

.github/workflows/build-deploy-pudl.yml

zaneselvans · 2023-12-19T01:27:19Z

.github/workflows/build-deploy-pudl.yml

+      - name: Tag nightly build
+        if: ${{ (github.event_name == 'schedule') }}
+        run: |
+          git config user.email "[email protected]"
+          git config user.name "PudlBot"
+          git tag -a -m "$NIGHTLY_TAG" $NIGHTLY_TAG $GITHUB_REF
+          git push origin $NIGHTLY_TAG
+


I think this is doing what we want it to now. It's a tag, it's applied to the branch that we're going to build (not main which is checked out here).

Sweet! Just to get this straight: we are tagging whatever this build gets kicked off on. Since this only triggers on schedule, and GITHUB_REF gets munged to dev in that case, we will always be tagging whatever the latest dev is with nightly-build-202x-xx-xx.

And then, when the nightly build succeeds, we can tell the nightly branch to point at nightly-build-202x-xx-xx and everyone who downloads the most recent nightly build off AWS will know that that corresponds to the nightly branch. But if they want to download nightly-build-202x-xx-xx from AWS they will also be able to associate that with the nightly-build-202x-xx-xx tag.

I think that's close, except that we only distribute the most recent nightly build and tagged releases to AWS. I imagine we would also have a copy of the most recent release in a stable/ directory. So if you want to refer to nightly-202x-xx-xx you would need to look at the gs://builds.catalyst.coop/nightly-202x-xx-xx path where the last 30 days of nightly build outputs are stashed, regardless of whether they were successful or not. So if you want to look at nightly build outputs + the code that generated them, you can look at the GCS output, and the tagged commit that matches its name.

Ah, right - that makes sense! Are there any barriers to distributing a few recent nightly builds on AWS as well as GCS?

We have a limited storage quota on AWS, and I don't think we actually want the public looking at our nightly build outputs, which may or or may not correspond to successful runs.

.github/workflows/docker-build-test.yml

…ease notes config.

zaneselvans

@jdangerx Tried to implement the merge-tag-into-branch thing here and cleaned up the Dockerfile a bit. My attempt to test the git authentication from within the container failed though.

.github/workflows/build-deploy-pudl.yml

.github/workflows/release.yml

docker/Dockerfile

docker/gcp_pudl_etl.sh

zaneselvans · 2023-12-20T06:11:51Z

I went ahead and created nightly and stable branches and a recently nightly-2023-12-19 tag where they would currently be, so that if this stuff runs it finds them.

docker/gcp_pudl_etl.sh

- Remove ad-hoc test of nightly build tagging - Use "$DOUBPLE_QUOTES" around envvars to prevent globbing/split words - Use $BUILD_ID consistently when referring to nightly build outputs. - Update name of epacems directory that we remove before deployment.

zaneselvans · 2023-12-20T19:50:18Z

@jdangerx Cleaned up the build script to remove ad-hoc tests. Integrated some of the bugfixes from #3171 here and extended them into the build script -- we construct a single $BUILD_ID in the workflow now and pass it in to the container, rather than reconstructing the ID in different ways in different places leading to broken URLs etc...

.github/workflows/build-deploy-pudl.yml

docker/gcp_pudl_etl.sh

jdangerx

Just one blocking question about the datasette build trigger. I think we can knock out like half of #3140 in one PR though once this gets merged!

.github/release.yml

docker/Dockerfile

docker/gcp_pudl_etl.sh

zaneselvans requested a review from jdangerx December 15, 2023 18:55

zaneselvans added the nightly-builds Anything having to do with nightly builds or continuous deployment. label Dec 15, 2023

zaneselvans requested a review from bendnorman December 15, 2023 18:55

zaneselvans self-assigned this Dec 15, 2023

zaneselvans mentioned this pull request Dec 15, 2023

Move to trunk-based development #3140

Closed

jdangerx reviewed Dec 18, 2023

View reviewed changes

.github/workflows/build-deploy-pudl.yml Outdated Show resolved Hide resolved

zaneselvans mentioned this pull request Dec 18, 2023

Automate the data release process #2756

Closed

zaneselvans force-pushed the tag-nightly-stable branch from 74e3ecf to b033044 Compare December 19, 2023 00:32

zaneselvans changed the title ~~Tag commits of nightly builds and last stable release.~~ Improve nightly build and deployment logic Dec 19, 2023

zaneselvans requested a review from jdangerx December 19, 2023 01:33

zaneselvans commented Dec 19, 2023

View reviewed changes

zaneselvans commented Dec 20, 2023

View reviewed changes

.github/workflows/docker-build-test.yml Outdated Show resolved Hide resolved

zaneselvans force-pushed the tag-nightly-stable branch from a946e01 to 7319bab Compare December 20, 2023 05:24

zaneselvans added 17 commits December 19, 2023 23:41

Update release.yml

06fda97

Tag commits of nightly builds and last stable release.

97a6089

Test pushing a tag in a GitHub action

e83334e

Add email and name to git user before pushing tag.

7cc0f18

Change envvar syntax to capture date in tag.

ebb594b

Push tag first, go faster.

b0f218e

Try to fix nightly tag workflow

9c597e9

Try to fix nightly tag workflow

1171321

Set up nightly tag in deploy workflow.

f42e951

Use git commands to tag stable during release.

736476e

Purge old files from distribution path before uploading new files.

afc25af

Apply nightly-YYYY-MM-DD to GIHUB_REF not main

273d173

Try to tag nightly from within docker container

fa0c044

Make nightly & stable branches. Safer removal of old deployments. Rel…

fe5e224

…ease notes config.

Copy repo contents into container before building conda env.

b68bb63

Attempt to clone repo inside Docker container

eef0c62

Attempt to clone repo inside Docker container

c0f1be7

zaneselvans added 8 commits December 19, 2023 23:41

Attempt to clone repo inside Docker container

85e9449

Revert to copying repo into container

16445d4

Move pudl repo into the pudl/ directory

ddfb861

Stop using conda run when not required.

9f98de6

Try and do a git thing

b3b3f52

Test git authentication with workflow_dispatch

f989635

Add some logging to git authentication check

5f1228b

Update settings file path to point at repo in container

b457942

zaneselvans force-pushed the tag-nightly-stable branch from 7319bab to b457942 Compare December 20, 2023 05:45

zaneselvans commented Dec 20, 2023

View reviewed changes

zaneselvans removed the request for review from bendnorman December 20, 2023 13:40

unset read-only git config

03f6e19

jdangerx reviewed Dec 20, 2023

View reviewed changes

docker/gcp_pudl_etl.sh Outdated Show resolved Hide resolved

zaneselvans added 3 commits December 20, 2023 13:39

Consolidate envvars and set explicit BUILD_ID

f409d8d

Clean up GCP PUDL ETL script

ac31f1e

- Remove ad-hoc test of nightly build tagging - Use "$DOUBPLE_QUOTES" around envvars to prevent globbing/split words - Use $BUILD_ID consistently when referring to nightly build outputs. - Update name of epacems directory that we remove before deployment.

remove old ACTION_SHA add BUILD_ID to container envvars

2e6097c

zaneselvans commented Dec 20, 2023

View reviewed changes

.github/workflows/build-deploy-pudl.yml Show resolved Hide resolved

zaneselvans commented Dec 20, 2023

View reviewed changes

.github/workflows/build-deploy-pudl.yml Show resolved Hide resolved

zaneselvans commented Dec 20, 2023

View reviewed changes

docker/gcp_pudl_etl.sh Show resolved Hide resolved

jdangerx requested changes Dec 20, 2023

View reviewed changes

.github/release.yml Outdated Show resolved Hide resolved

docker/Dockerfile Outdated Show resolved Hide resolved

docker/Dockerfile Show resolved Hide resolved

docker/gcp_pudl_etl.sh Show resolved Hide resolved

Reinstate caching of conda environment in Docker image.

de58851

jdangerx approved these changes Dec 20, 2023

View reviewed changes

zaneselvans marked this pull request as ready for review December 20, 2023 21:10

zaneselvans enabled auto-merge (squash) December 20, 2023 21:18

zaneselvans merged commit a0dcdf3 into main Dec 20, 2023
14 checks passed

zaneselvans deleted the tag-nightly-stable branch December 20, 2023 22:47

zaneselvans mentioned this pull request Dec 20, 2023

Restructure build-deploy action #3171

Closed

jdangerx mentioned this pull request Dec 20, 2023

Remove references to dev branch in preparation for deprecation #3181

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve nightly build and deployment logic #3164

Improve nightly build and deployment logic #3164

zaneselvans commented Dec 15, 2023 •

edited

Loading

jdangerx left a comment

zaneselvans left a comment

zaneselvans Dec 19, 2023

jdangerx Dec 19, 2023

zaneselvans Dec 19, 2023

jdangerx Dec 20, 2023

zaneselvans Dec 20, 2023

zaneselvans left a comment

zaneselvans commented Dec 20, 2023

zaneselvans commented Dec 20, 2023

jdangerx left a comment

Improve nightly build and deployment logic #3164

Improve nightly build and deployment logic #3164

Conversation

zaneselvans commented Dec 15, 2023 • edited Loading

Overview

Issues / Questions:

jdangerx left a comment

Choose a reason for hiding this comment

zaneselvans left a comment

Choose a reason for hiding this comment

zaneselvans Dec 19, 2023

Choose a reason for hiding this comment

jdangerx Dec 19, 2023

Choose a reason for hiding this comment

zaneselvans Dec 19, 2023

Choose a reason for hiding this comment

jdangerx Dec 20, 2023

Choose a reason for hiding this comment

zaneselvans Dec 20, 2023

Choose a reason for hiding this comment

zaneselvans left a comment

Choose a reason for hiding this comment

zaneselvans commented Dec 20, 2023

zaneselvans commented Dec 20, 2023

jdangerx left a comment

Choose a reason for hiding this comment

zaneselvans commented Dec 15, 2023 •

edited

Loading