Avoid submitting packages to Factory that we haven't actually tested yet #352
Conversation
There was a problem hiding this comment.
So if I get it correctly, we have one staging place (testing) which will be used by the oQA-in-oQA and if the tests pass, the packages will be "released" to the tested for the automated submission into Factory, otherwise it will be cleanup upon failure to make space for next (I guess Jenkins) run to copy the code from the devel repo to testing for the actual testing. If we manage to make multiple changes and there is still something in the staging, it will be hold for a moment and batched together in the next opportunity, right? And the current release process does not need any change, because we only changed the way how to get the packages into the "tested" space...
I do have one question inline, otherwise it looks good to me.
Not exactly. I am quoting "release" here because this is how the osc command is called. What we are actually doing here is creating a snapshot for the sake of testing and creating a submit request if tests pass.
Yes, we're talking about hourly scheduled pipeline runs on Jenkins here.
That will effectively happen. However, nothing is ever explicitly batched up. If there are still pending tests (or the submission itself is still pending) when Jenkins runs the next trigger job this trigger job will simply return early and do nothing. One hour later there will hopefully no pending tests/submissions anymore so we can trigger tests again (with all the changes that have been made meanwhile).
Yes, there's no additional step involved (that needed to be configured in Jenkins). |
Martchus
force-pushed
the
avoid-submitting-untested-packages
branch
from
40ef775 to
5372e66
Compare
|
Thank you for the explanation. Yes, we are on the same page here. |
Martchus
force-pushed
the
avoid-submitting-untested-packages
branch
from
5372e66 to
af3b461
Compare
The packages on `devel:openQA` might change while openQA-in-openQA tests are executing. So far we submit packages directly from `devel:openQA` after openQA-in-openQA have passed. This means we don't neccassarily submit the version of the packages we have actually tested. With this change we "release" a snapshot of `devel:openQA` to the staging repository `devel:openQA:testing` when triggering tests and use that within openQA-in-openQA tests (see os-autoinst/os-autoinst-distri-openQA#215). We then only submit from this staging repository when tests have passed. This way we only submit what we've tested despite changes on `devel:openQA`. Packages in the staging repository are cleaned up when openQA-in-openQA tests fail or after packages have been submitted. This trigger script will only release a new version into the staging project if it has been cleaned up ensuring its contents stay the same while tests/submissions are pending. This commit also adds the new script `cleanup-obs-project`. It conveniently deletes all packages in an OBS project without deleting the project itself. We might want to add it on Jenkins as post-failure action for monitoring and submissions to avoid leftovers. (It is also useful to test the deletion code on its own.) Related ticket: https://progress.opensuse.org/issues/167395
Martchus
force-pushed
the
avoid-submitting-untested-packages
branch
from
af3b461 to
c92befb
Compare
|
I suppose it would be best if we merge this only after hackweek. There might be further changes required we'll only discover when it actually runs on Jenkins. |
The packages on
devel:openQAmight change while openQA-in-openQA tests are executing. So far we submit packages directly fromdevel:openQAafter openQA-in-openQA have passed. This means we don't neccassarily submit the version of the packages we have actually tested.With this change we "release" a snapshot of
devel:openQAto the staging repositorydevel:openQA:testingwhen triggering tests and use that within openQA-in-openQA tests (seeos-autoinst/os-autoinst-distri-openQA#215). We then only submit from this staging repository when tests have passed. This way we only submit what we've tested despite changes on
devel:openQA.Packages in the staging repository are cleaned up when openQA-in-openQA tests fail or after packages have been submitted. This trigger script will only release a new version into the staging project if it has been cleaned up ensuring its contents stay the same while tests/submissions are pending.
This commit also adds the new script
cleanup-obs-project. It conveniently deletes all packages in an OBS project without deleting the project itself. We might want to add it on Jenkins as post-failure action for monitoring and submissions to avoid leftovers. (It is also useful to test the deletion code on its own.)Related ticket: https://progress.opensuse.org/issues/167395
I tested this locally by running the individual added/changed parts of the scripts. I tested whether the osc commands to what they are supposed to against the real OBS instance where the project https://build.opensuse.org/project/show/devel:openQA:testing already exists.
I haven't adjusted any pipelines on Jenkins yet. I suppose this is also not necessary. It might be nice invoke
cleanup-obs-projectas post-failure action on http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW/configure and http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/configure. One cannot configure post-failure actions explicitly but I suppose we can just append an|| …in the shell script.