Add a workflow building nightlies #195

webknjaz · 2020-09-30T18:02:11Z

This is a workflow roughly copied from https://github.com/webknjaz/ansinight.

Ref: #98

webknjaz · 2020-09-30T18:06:08Z

It worked in my fork: https://webknjaz.github.io/antsibull/

abadger · 2020-09-30T18:13:06Z

The major question I have is where should the nightly artifacts live? They don't quite fit within antsibull since antsibull is more the build tool to create the artifacts. I think we have a generic need for "someplace where files, documentation, communication, tickets" related to the project as a whole should live and we should put everything there.

Going to put this question on the community irc meeting agenda.

webknjaz · 2020-09-30T18:39:32Z

@abadger I suggest merging this as is and then changing when the decision is available. This way, we'll at least have some build flow smoke-testing.

webknjaz · 2020-09-30T18:41:14Z

.github/workflows/build-dumb-pypi.yml

+  workflow_dispatch:
+    inputs:
+      ansible-base-slug:
+        default: ansible/ansible
+        description: The GitHub repo slug for pulling ansible-base from
+        required: true
+      ansible-base-ref:
+        default: devel
+        description: The GitHub repo ref for pulling ansible-base from
+        required: true
+      ansible-build-data-slug:
+        default: ansible-community/ansible-build-data
+        description: The GitHub repo slug for pulling ansible build data from
+        required: true
+      ansible-build-data-ref:
+        default: main
+        description: The GitHub repo ref for pulling ansible build data from
+        required: true


@abadger look at this. This is experimental and is supposed to add a button allowing you to trigger a workflow from the GitHub UI. I haven't tested it but it should appear after the merge.

Ref: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

@abadger this specific piece of setup could be used to automate generating (and potentially publishing) the stable releases to PyPI. Do you have any opinion on this? IIRC the build happens by manual runs locally, right?

I do want to have a push button to make a release eventually. There's one piece of coordination involved in making a release right now, though, updating the docsite. Possibilities:

Make the build one push button and publish a second one. That way the release manager can alert the docs team in between.

Ask the docs team if they can update the docs after we've published. Now that the docs build is back down to ~ ten minutes I think they might be okay with that.

Note that building and publishing includes saving information about the build/release to the ansible-build-data repository. and has slightly different means of workflows depending on what type of release it is (alpha, beta, rc, patch release). So the script that generates the new release tarballs will need to take some input to tell what sort of release it was and also be able to commit to the ansible-build-data repo. I also found that I needed to make a special release today and so I ran a slightly different command to do that. Special things might just be special, though. It just means that we should make sure that it's easy for people to understand what the script does so that they can figure out how to do special, one-off things. The build instructions are here: https://github.com/ansible-community/antsibull/blob/main/docs/build-ansible.rst

I'm not sure I understand how it's bound to docs: can't this happen asynchronously? Also, building the docs can happen automatically if it's something the docs team is up to: an automatic workflow can send notifications to other systems, in particular, it can trigger workflows in other repos. Does the publish strictly depend on docs?

abadger · 2020-10-28T19:44:21Z

This was mentioned in today's community irc meeting. There's a desire for an ansible tarball that can be used with devel builds of ansible-base.

.github/workflows/build-dumb-pypi.yml

gundalow · 2020-11-05T10:22:59Z

As agreed in yesterdays Community Meeting this will live in https://github.com/ansible-community/nightly-builds

Thanks for doing this :)

webknjaz · 2020-11-05T11:52:35Z

@gundalow @abadger we've decided that the static DumbPyPI website will live @ ansible-community/nightly-builds. But do you have any preferences wrt the CI/CD workflow itself? I'm pretty sure that I can set it up to push the GH Pages content to another repo while keeping the workflow in antsibull if that's what people want.

gundalow · 2020-11-05T11:57:01Z

@gundalow @abadger we've decided that the static DumbPyPI website will live @ ansible-community/nightly-builds. But do you have any preferences wrt the CI/CD workflow itself? I'm pretty sure that I can set it up to push the GH Pages content to another repo while keeping the workflow in antsibull if that's what people want.

I don't mind either way.

.github/workflows/build-dumb-pypi.yml

webknjaz · 2020-11-05T12:04:46Z

.github/workflows/build-dumb-pypi.yml

+        pip setuptools wheel
+    - name: Install antsibull dist from PyPI
+      run: python -m pip install -U --user antsibull
+    - name: Install antsibull dist from GitHub


@gundalow @abadger this currently uses the bleeding-edge for antsibull from Git rather than getting it from PyPI. This may be a reason to keep the workflow here + it'll update the index when the antsibull repo itself gets updates...

I was thinking that it should be put into the nightly-build repository (keep the artifacts and the tools close together) until this. I do wonder if the nightly build should use antsibull from git, though, or stick with the antsibull on pypi?

After I gave it some thought, I decided to try having it here because the builder is here too. Also, when I was first trying to make it work, PyPI version didn't have all the necessary bits for some reason and I needed something unreleased. I think if we keep it here, it'll also test that antsibull's builder is healthy: it seems reasonable to ensure that it doesn't break with some random change to antsibull.

Of course, we can move it later if it proves to be too annoying.

This would also support goals declared in #98 — how can we build a release automation if the tooling does not test this case?

I think testing should be in integration and unit testing. Anything that results in artifacts that the public can access should be considered part of production infrastructure instead.

I would agree for the regular tagged releases that are based on a human sign-off but nightlies are supposed to be made out of potentially broken things and I consider them a part of smoke and integration test infrastructure. OTOH if the tool breaks on nightlies, it's not the end of the world but is still a strong signal that something needs to be fixed... To me, it sounds better than finding out that it doesn't work right before making a release.

To be clear, as long as we're using a git checkout of the antsibull git repo, I think it is good to host it here. I'm just not sure if we want to keep doing that down the road.

.github/workflows/build-dumb-pypi.yml

abadger · 2020-11-05T14:59:58Z

Heh, if you wanted sign off from me, it would have been nice to wait until I was awake ;-)

webknjaz · 2020-11-05T15:03:31Z

Sorry, I first wanted to ask for opinions but then did some thinking and realized that this change is non-destructive and would need follow-up updates anyway :-)

abadger · 2020-11-05T15:13:57Z

A couple things for the followup:

Would be nice to drop a README in the pypi repo that says:
- What the repos purpose is intended for and what it is not intended for (like samccann's note yesterday that the repo should say that it is bleeding edge, not to be used in production, mirrors the active development branch, etc)
- Where to find the script which creates the repository.
- We probably want the information both in the toplevel README of the github repo since that's the standard place for people to browse when trying to find information about a github project and in the github page https://ansible-community.github.io/nightly-builds/
Maybe we should rename the workflow (file and toplevel name) since we may be able to host the Github action to built the releases here as well. Something that mentions this is for the nightlies I think?

abadger · 2020-11-05T15:15:19Z

Oh, and the title on https://ansible-community.github.io/nightly-builds/ should probably be changed from "My Private PyPi" as well.

webknjaz · 2020-11-05T15:25:23Z

Makes sense, I'll keep that in mind for the next PR.

webknjaz force-pushed the maintenance/nightlies branch from e065c80 to 32fde41 Compare September 30, 2020 18:04

abadger mentioned this pull request Sep 30, 2020

Community Working Group Meeting Agenda ansible/community#539

Closed

webknjaz force-pushed the maintenance/nightlies branch from 32fde41 to ca6265c Compare September 30, 2020 18:38

webknjaz commented Sep 30, 2020

View reviewed changes

webknjaz force-pushed the maintenance/nightlies branch from ca6265c to 191f3ef Compare September 30, 2020 18:46

dmsimard reviewed Nov 4, 2020

View reviewed changes

.github/workflows/build-dumb-pypi.yml Show resolved Hide resolved

webknjaz commented Nov 5, 2020

View reviewed changes

.github/workflows/build-dumb-pypi.yml Show resolved Hide resolved

webknjaz commented Nov 5, 2020

View reviewed changes

.github/workflows/build-dumb-pypi.yml Outdated Show resolved Hide resolved

webknjaz force-pushed the maintenance/nightlies branch from 191f3ef to b64adf5 Compare November 5, 2020 13:21

Add a workflow building nightlies

ffa9b71

webknjaz force-pushed the maintenance/nightlies branch from b64adf5 to ffa9b71 Compare November 5, 2020 13:22

webknjaz merged commit 47524ca into ansible-community:main Nov 5, 2020

webknjaz deleted the maintenance/nightlies branch November 5, 2020 14:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a workflow building nightlies #195

Add a workflow building nightlies #195

webknjaz commented Sep 30, 2020 •

edited

Loading

webknjaz commented Sep 30, 2020

abadger commented Sep 30, 2020

webknjaz commented Sep 30, 2020

webknjaz Sep 30, 2020

webknjaz Nov 5, 2020

abadger Nov 5, 2020

webknjaz Nov 5, 2020

abadger commented Oct 28, 2020

gundalow commented Nov 5, 2020

webknjaz commented Nov 5, 2020

gundalow commented Nov 5, 2020

webknjaz Nov 5, 2020

abadger Nov 5, 2020

webknjaz Nov 5, 2020

webknjaz Nov 5, 2020

abadger Nov 5, 2020

webknjaz Nov 5, 2020

abadger Nov 5, 2020

abadger commented Nov 5, 2020

webknjaz commented Nov 5, 2020

abadger commented Nov 5, 2020

abadger commented Nov 5, 2020

webknjaz commented Nov 5, 2020

Add a workflow building nightlies #195

Add a workflow building nightlies #195

Conversation

webknjaz commented Sep 30, 2020 • edited Loading

webknjaz commented Sep 30, 2020

abadger commented Sep 30, 2020

webknjaz commented Sep 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abadger commented Oct 28, 2020

gundalow commented Nov 5, 2020

webknjaz commented Nov 5, 2020

gundalow commented Nov 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abadger commented Nov 5, 2020

webknjaz commented Nov 5, 2020

abadger commented Nov 5, 2020

abadger commented Nov 5, 2020

webknjaz commented Nov 5, 2020

webknjaz commented Sep 30, 2020 •

edited

Loading