feat: create nightly builds #369

iand · 2021-07-22T11:30:37Z

Closes #363

⚠️ Merge this after #367

High level summary

This PR adds a cron job that runs every day and builds nightly releases for each package name in ./dists
nightly.yml workflow is built on top of modernized main.yml (feat: macos signing and notarization #367) and runs the same builds scripts
- building 23+ packages takes time, so we build them separately, in parallel – each one having own isolated output proving it can be built end-to-end
- global status can be quickly inspected on https://github.com/ipfs/distributions/actions/workflows/nightly.yml and clicking on the latest run allows further inspection https://github.com/ipfs/distributions/actions/runs/1077161039
- 👉 If one of the packages fails to build, workflow will create a new issue (or comment on existing open one) – examples: Nightly build failed for ipfs-pack #374 & Nightly build failed for ipfs-see-all #375

Build scripts changes

./dist.sh add-version <dist> nightly will add a nightly version to the versions file using the current date (@lidel is this still useful?)
make nightly add a nightly version to every dist versions file using the current date
- one can also cd dists/<package> && make nightly to add it only to specific package - we use it on CI to run builds in paralllel (@lidel)
make dist will build nightly versions by checking out the latest version of the default branch and producing a build artifact named using the pattern nightly-yyyy-mm-dd-hash where hash is the starting 7 (at least) characters of the commit hash. It may be more than 7 if a shorter length would be ambiguous.
make all_dists is unchanged but will build a nightly version in the course of running make dist above
It's not expected that the nightly-yyyy-mm-dd version will be committed to the repo. It's an ephemeral version that is generated to produce a nightly build.
The final versions file is normally added to ipfs my make publish. This will now include the most recent nightly version. This allows the build system to ignore it on subsequent runs until a new day is reached.

TODO

wire up ability to build nightly-* versions using existing make publish pipeline (@iand)
add publish step to nightly.yml (@iand)
do not pollute ./nightly (@lidel)
ensure build does not use commits older than the nightly threshold (@lidel)
TODO: explore checking out the last commit in the default branch before arbitrary date: git rev-list -1 --first-parent --before="2021-07-27" "$(git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@')" – makes nightlies more deterministic (at the revision level)
fix vtag releases (like fs-repo-migrations/fs-repo-0-to-1) (@lidel)
share build scripts with main.yml (@lidel)
refactor to include signing (or make part of main.yml
- extracted to Make nightlies signed #378

common.mk

dist.sh

.github/workflows/nightly.yml

iand · 2021-07-22T15:14:27Z

This doesn't handle any specific publishing logic for nightlies. I haven't tested but I expect that one nightly build for each day will be published.

How do we want to handle clean up of old nightlies? What is the best mechanism for keeping track of how many nightlies have been built so we can restrict to a given number, e.g. 7.

lidel · 2021-07-22T16:44:33Z

Number of nightlies per package can be calculated by grep nightly ./dists/{name}/versions | wc -l

I imagine dropping of the oldest nightlies over the limit could happen in scripts/patch.js (executed during make publish) after everything is patched, but right before returning the final CID.

iand · 2021-07-23T10:41:48Z

Number of nightlies per package can be calculated by grep nightly ./dists/{name}/versions | wc -l

No, since versions will contain only the most recent nightly since the nightly CI does not commit updates to the versions file back to the repo. See 4th bullet in the pr description.

I think the simplest place to do it is in build-go.sh as an additional step (5) in the existing sequence:

check out repo ,dists/{name}/versions just contains tagged versions
append nightly version to dists/{name}/versions
compare with /ipfs/Qm.../{name}/versions and build anything new
copy dists/{name}/versions to releases/{name}/versions
append nightly builds from /ipfs/Qm.../{name}/versions to releases/{name}/versions (keeping 7)
publish releases/{name}/versions as /ipfs/Qm.../{name}/versions which will now contain tagged versions and 7 nightly

I imagine dropping of the oldest nightlies over the limit could happen in scripts/patch.js (executed during make publish) after everything is patched, but right before returning the final CID.

this tests build without any new releases

#367 (comment)

This is a small refactor that makes sure we only add meaningful files to ./releases dir (it is important for signing etc)

lidel

Works as expected, thank you, @iand! 🙌

I will take this over and tweak it a bit to be compatible with final approach from #367
(will comment when ready)

This ensures that "nightly-2020-01-01" won't have any commits that happened on "2020-01-02" – enabling us to build historical nightlies.

this makes nightly do exactly what release build does, but allows us to tweak some behaviors, for example do pins that expires after 7 days, or report each preview separately

This runs cleanup only when we have anything to clean up

lidel · 2021-07-29T12:29:13Z

This is ready for review, but should be merged after feat: macos signing and notarization #367
nightly.yml workflow is built on top of the modernized main.yml (feat: macos signing and notarization #367) and runs the same builds scripts
- it does not do signing of produced nighties yet because of macOS worker limits, but we can add it in follow-up PR
- building 23+ packages takes time, so we build them separately, in parallel – each one having own isolated output prooving it can be built end-to-end
- global status can be quickly inspected on https://github.com/ipfs/distributions/actions/workflows/nightly.yml and clicking on the latest run allows further inspection https://github.com/ipfs/distributions/actions/runs/1077161039
- 👉 If one of the packages fails to build, workflow will create a new issue (or comment on existing open one) – examples: Nightly build failed for ipfs-pack #374, Nightly build failed for ipfs-see-all #375

lidel

Ready for review.

lidel · 2021-07-29T12:40:30Z

.github/workflows/nightly.yml

+on:
+  push:
+  ## Allow manual invocation
+  #workflow_dispatch:
+  #schedule:
+  #  - cron: '0 5 * * *' # UTC


ℹ️ we will remove push and restore schedule for cron trigger + workflow_dispatch for running nightly builds on-demand

lidel · 2021-07-29T12:54:31Z

.github/workflows/nightly.yml

+      - run: cd ./dists/${{ matrix.dist_name }} && make nightly
+      - run: ./dockerized make publish
+      - name: Create issue if build failed
+        uses: actions/github-script@v4


ℹ️ Third-party turn-key solutions exists, but I decided to give actions/github-script a try. It is an official action provided by GitHub, provides JS porcelain for running arbitrary API calls.

Debugging is a bit painful, but being written in JS could make it more dev friendly than bash+curl, allowing more people to maintain/extend. actions/github-script is extremely powerful, only caveat is that it needs a personal access token to do anything outside the current repo.

We may leverage it even more for developer flow automation tailored to our needs. For example, it could automatically create/label issues and add them to specific column in specific project.

lidel · 2021-07-29T12:55:19Z

.github/workflows/nightly.yml

+        env:
+          PIN_CID: ${{ steps.cid-reader.outputs.CID }}
+          PIN_NAME: "ipfs/distributions/nightly/${{ matrix.dist_name }}"
+          PIN_ADD_EXTRA_ARGS: "--expire-in 168h"


ℹ️ this pins nightly artifacts only for a week

BigLep

Great work!
One small idea is to consider the comments you left in the PR to inline them into the code for discoverability.

lidel and others added 2 commits July 21, 2021 21:31

wip: signing poc with hardcoded dist details

cd78bdf

feat: create nightly builds

230d1cd

lidel reviewed Jul 22, 2021

View reviewed changes

common.mk Outdated Show resolved Hide resolved

dist.sh Outdated Show resolved Hide resolved

ipfs deleted a comment from welcome bot Jul 22, 2021

Ensure nightly dates use utc

1bb2f20

iand commented Jul 22, 2021

View reviewed changes

.github/workflows/nightly.yml Outdated Show resolved Hide resolved

Allow manual invocation of nightly workflow

7208eae

BigLep linked an issue Jul 23, 2021 that may be closed by this pull request

Create nightly builds #363

Closed

13 tasks

iand and others added 4 commits July 23, 2021 12:16

Keep last 7 nightly versions in published versions file

42531b6

Touch existing versions file to create it

18ca56e

Create output dir earlier

4870454

wip: debugging pinning to cluster

8e11abe

iand marked this pull request as ready for review July 26, 2021 09:51

Publish after nightly build

2cb0822

iand requested a review from lidel July 26, 2021 10:04

lidel added 8 commits July 26, 2021 14:26

wip: wait for 2 cluster confirmations

f3c456d

wip: speed up build

3e9a8e2

wip: PR preview link

796aaae

wip: no-op build

ed656e5

this tests build without any new releases

refactor: use GO_IPFS_VER in dockerized build

7d222b5

docs: DNSLink discussion

f115f93

refactor: simplify

9acff7e

#367 (comment)

refactor: avoid polluting ./releases

5ea24dc

This is a small refactor that makes sure we only add meaningful files to ./releases dir (it is important for signing etc)

lidel reviewed Jul 27, 2021

View reviewed changes

lidel self-assigned this Jul 27, 2021

lidel added 2 commits July 27, 2021 19:16

refactor: respect nightly cut-off date

5df807b

This ensures that "nightly-2020-01-01" won't have any commits that happened on "2020-01-02" – enabling us to build historical nightlies.

wip: see how long all nightlies take to build and pin

716ebaf

lidel added 2 commits July 28, 2021 16:28

refactor: github-preview-link.sh

835df84

test: signed release built for [email protected]

1c9b839

lidel changed the base branch from master to feat/signed-ci-build July 28, 2021 15:45

lidel added 4 commits July 28, 2021 17:52

Merge branch 'feat/signed-ci-build' into feat/issue-363-nightlies

00dc19b

refactor: reuse ./scripts/ci

e2570b8

this makes nightly do exactly what release build does, but allows us to tweak some behaviors, for example do pins that expires after 7 days, or report each preview separately

fix: nightly cleanup

947f37d

This runs cleanup only when we have anything to clean up

refactor: deduplicate pinning

5277083

lidel force-pushed the feat/issue-363-nightlies branch 12 times, most recently from ae80100 to db09ba3 Compare July 29, 2021 02:54

feat: fill issue for every broken nightly build

843feda

lidel force-pushed the feat/issue-363-nightlies branch from db09ba3 to 843feda Compare July 29, 2021 03:08

fix: github status link

08caa11

lidel approved these changes Jul 29, 2021

View reviewed changes

BigLep reviewed Jul 29, 2021

View reviewed changes

Base automatically changed from feat/signed-ci-build to master August 3, 2021 22:44

lidel added 2 commits August 4, 2021 00:49

Merge branch 'master' into feat/issue-363-nightlies

dc8b64e

chore: restore target trigger (cron and manual)

6a29155

lidel merged commit 55b8923 into master Aug 3, 2021

lidel deleted the feat/issue-363-nightlies branch August 3, 2021 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: create nightly builds #369

feat: create nightly builds #369

iand commented Jul 22, 2021 •

edited by lidel

Loading

iand commented Jul 22, 2021

lidel commented Jul 22, 2021 •

edited

Loading

iand commented Jul 23, 2021 •

edited

Loading

lidel left a comment

lidel commented Jul 29, 2021 •

edited

Loading

lidel left a comment

lidel Jul 29, 2021

lidel Jul 29, 2021

lidel Jul 29, 2021

BigLep left a comment

feat: create nightly builds #369

feat: create nightly builds #369

Conversation

iand commented Jul 22, 2021 • edited by lidel Loading

⚠️ Merge this after #367

High level summary

Build scripts changes

TODO

iand commented Jul 22, 2021

lidel commented Jul 22, 2021 • edited Loading

iand commented Jul 23, 2021 • edited Loading

lidel left a comment

Choose a reason for hiding this comment

lidel commented Jul 29, 2021 • edited Loading

lidel left a comment

Choose a reason for hiding this comment

lidel Jul 29, 2021

Choose a reason for hiding this comment

lidel Jul 29, 2021

Choose a reason for hiding this comment

lidel Jul 29, 2021

Choose a reason for hiding this comment

BigLep left a comment

Choose a reason for hiding this comment

iand commented Jul 22, 2021 •

edited by lidel

Loading

lidel commented Jul 22, 2021 •

edited

Loading

iand commented Jul 23, 2021 •

edited

Loading

lidel commented Jul 29, 2021 •

edited

Loading