rework gangplank and pipelines

[dustymabe]

Currently the only known "productized" use of gangplank that I know of is in our multi-arch builds for Fedora CoreOS. Occasionally we have issues that need investigating and we currently have no one on the team that really knows the code base well. Since we're only using one of the most basic features of gangplank and we don't have maintainers or even semi-experts on the code base I suggest we supplant the use gangplank with just using podman --remote directly for our uses.

I've done some basic testing and I think we can get everything we want from podman --remote. Thought I would test the waters here to see what other people think of the direction. WDYT?

[miabbott]

@sosiouxme I believe you had hopes for gangplank for orchestrating multi-arch builds

[cgwalters]

The way I think of Gankplank is ~three things:

1. Basically implementing podman --remote again
2. Better supporting splitting up build components into separate kube pods
3. Trying to move the codebase more towards Go (I filed that separately as Make programming language default to Go #2821 )

I agree with you that for what the FCOS pipeline is doing now, directly using podman --remote seems much better.

Now of the above I think 2. is still relevant. However...there's always been a huge tension between that and e.g. Prow which also wants to be the thing that spawns pods. @cheesesashimi has done some work today on using Prow to shard test pods for example. The Jenkins pipelines also spawn pods.

It may be that this aspect of gangplank should remain.

[travier]

Now that we are on OCP 4.x for all clusters, we might also want to evaluate using Pipelines / Tekton for our CI:

• https://docs.openshift.com/container-platform/4.10/cicd/pipelines/understanding-openshift-pipelines.html
• https://tekton.dev/

[mike-nguyen]

Now that we are on OCP 4.x for all clusters, we might also want to evaluate using Pipelines / Tekton for our CI:

* https://docs.openshift.com/container-platform/4.10/cicd/pipelines/understanding-openshift-pipelines.html

* https://tekton.dev/

Our multi-arch pipelines are still OCP 3.x

[ravanelli]

A huge amount of effort has ben done in the gangplank implementation, it feels like sad to drop it entirely.

I tried to summarize where we are today here. Nonetheless, I agree with @dustymabe, we still need more improvements in order to use it, specially in the cases where the builds finish as success and we can see errors (kola tests and others) + more log infos that are not fully returned. For the fcos case I don't see any issue in replacing it with podman remote, if we can't fix these issues right away.
But for sure, if we still want to keep/use gangplank around, more work needs to be done, and more knowledge about the code needs to be learned.

[cverna]

I think a good question to answer is: Is this going to be the best use of our time ? Based on the previous replies I would lean on replying No to that question. While gangplank is a neat idea, I am not 100% sold that it fits our team's mission, we can only focus on a limited set of problems to solve and moving away from gangplank is giving us an opportunity to refocus elsewhere.

[sosiouxme]

The appeal of gangplank for ART is to enable more customization at the job spec level (something declarative, more flexible than job parameters, but not as low-level as COSA). For at least three use cases:

1. regular old production builds (with or without bootimages)
2. embargoed builds (send the output to private storage)
3. hotfixes (CoreOS builds with tweaks targeted to a single customer)

Exploration stalled again because we're easily distracted, but we still have a business need to do all these things. If not gangplank, then what? I don't know anything about the podman remote approach you mention. Of course as much as possible we want to be using well-supported tooling. If ART became the only user of gangplank that's probably not tenable.

[cgwalters]

hotfixes (CoreOS builds with tweaks targeted to a single customer)

Doesn't ocp coreos layering generally obsolete this, particularly once we support e.g. kernel overrides?

[dustymabe]

We are now no longer running gangplank tests in CI (#2895). In my opinion we can move forward with exploring an alternative for our specific needs and eventually remove gangplank once that is proven out.

Please speak up if you disagree.

[darkmuggle]

It become clear to me that Gangplank never meet the vision and there was resistance to the idea anyway. It was a fun project, but ultimately if it is not maintained and not useful, removing it seems to be the sensible path.

[cgwalters]

I don't think of this as removing gangplank, as I said above for example I think pushing the build system to be Go makes total sense and as part of #2919 I was reading through and looking at the gangplank code there. Conceptually that sub-thread is like "move parts of gangplank code into cosa toplevel", not get rid of it!

Also I definitely like the opinionated flow for generating kube pods. I think we should try to elevate that part of gangplank too to the toplevel. A tricky part has been intersecting that with Prow...what we ended up with today (raw cosa, not using gangplank) for this is honestly very hacky and it would be much more elegant with a gangplank-like approach.

This issue started with more "let's use podman --remote directly" which is a bit of what gangplank is doing, but not all of it. So...I took the liberty of just retitling this issue to reflect that.

[cgwalters]

strawman proposal:

• Finish off Make programming language default to Go #2821 i.e. Make /usr/bin/coreos-assembler a Go program, implement clean in Go #2919
• move gangplank code to github.com/coreos/gangplank
• switch to vendoring gangplank, and /usr/bin/gangplank -> /usr/bin/cosa

This would make it easier for us to basically incrementally merge the two things (cosa and gangplank) that are today somewhat artificially distinct. (I also think that writing new code doing container stuff in Python is the wrong move, and this helps counter that)

[dustymabe]

It become clear to me that Gangplank never meet the vision and there was resistance to the idea anyway. It was a fun project, but ultimately if it is not maintained and not useful, removing it seems to be the sensible path.

I don't think the problem was that Gangplank didn't "meet the vision". The problem is more that (at least for FCOS) we were never able to gain access to non x86_64 kube/openshift clusters so we were never able to take advantage of the features of gangplank. We have been using the gangplank "podman remote" mode for multi-arch builds, but that mode is less valuable than the other modes of gangplank and we can use podman remote direclty.

[cgwalters]

Yeah, I understand. But OTOH, while apparently Fedora Infra only has x86_64, RHEL/RHCOS does have OCP clusters with all target architectures. I hope that the Fedora Infra thing is going to be fixed (as we've discussed outside of this issue) - there can't be any blockers to using OCP on aarch64 in e.g. AWS for Fedora right?

Anyways...I think we're agreeing, or at least not disagreeing. It makes sense to use podman --remote directly. But...I am not so sure it makes sense to write nontrivial wrappers around that in Python versus Go, and specifically making use of the Go connections to the container ecosystem that we already have in gangplank code.

[travier]

We won't be able to have OCP clusters for all architectures for RHCOS in the future AFAIK, only individual nodes.

[cgwalters]

We won't be able to have OCP clusters for all architectures for RHCOS in the future AFAIK, only individual nodes.

I can't imagine that's true; I mean, simply to even test what we're shipping we need to run clusters...

[cgwalters]

OK new tweaked proposal; after #2979 lands and fedora-coreos-pipeline switches over to it vs gangplank, we:

• change the build machinery to stop building it, but keep the code for now because it feels likely to me we are going to find bits of code there we want to continue using. And I'll keep advocating for supporting something like COREOS_ASSEMBLER_REMOTE_SESSION=$(cosa remote-session --create-from-kubeconfig) and also supporting generating pods instead of using podman --remote
• move all the vendor/ bits to the toplevel (addressing add coreos-assembler remote-session command #2979 (comment) and workflows: add golangci-lint check #2978 (comment) )

[dustymabe]

I'm OK with leaving the code around, but I'd really like to not get alerts (i.e. dependabot) about things that need updating for it if we're not using it. If we can get away from that then I'm happy.

[cgwalters]

Yep; by moving the vendor directory to the toplevel, we will drop out everything that's not used by cosa/mantle. (And then, hopefully shortly after, make mantle into a Go library too so we get down to one vendor directory)

[dustymabe]

Yep; by moving the vendor directory to the toplevel, we will drop out everything that's not used by cosa/mantle.

I think what you mean by that is:

• move the mantle/vendor directory to the toplevel (merging with the now existing toplevel vendor dir)
• delete the gangplank/vendor directory and remove steps to build gangplank

Is that right?

[dustymabe]

Some work on this front to remove our use of gangplank in the FCOS pipeline:

• add coreos-assembler remote-session command #2979
• refactor pipelines to use cosa remote-session; drop gangplank fedora-coreos-pipeline#567

Once that second one merges we'll no longer be using gangplank for FCOS and I don't think there are any other current uses.

[dustymabe]

With #2979 and coreos/fedora-coreos-pipeline#567 our pipelines have now been converted to using podman --remote directly for multi-arch builds.

As of #3001 gangplank is no longer used or built, but the code is still in the code base for future reference/use.

We got rid of the vendor directory so all vendored deps have also been dropped.

I'm thinking this can be closed now.