CI to test package install and activation against versions/architectures of Pulsar

[gnmerritt]

Have you checked for existing feature requests?

• Completed

Summary

I ran into a couple packages not loading (activate throwing an error, see #580) when I tried to use the Apple Silicon build of Pulsar. I imagine that many more packages (any with natively compiled dependencies) are similarly broken on M1 macs.

It'd be great to have a CI system that tries to install a cross product of packages, versions architectures and verify that they can at least activate successfully. I imagine we could start with an allow-list and expand it as desired to cover more popular packages.

Could also only start with packages that have tests and no external dependencies.

I'd be happy to build this if folks think it's a reasonable idea.

What benefits does this feature provide?

This will help catch broken packages in the wake of the Atom domain sunset, and could allow the package repository to provide badging for "known to work with" versions / architectures.

It'd also provide a framework for evaluating the intrusiveness of any proposed breaking API changes

Any alternatives?

Another option would be to try to foist this work off on the package authors at build/publish time rather than centralizing it. I think that could also make sense but wouldn't provide as much visibility for pre-release versions as a central CI system would.

Other examples:

I believe Rust has something similar where they test crates.io packages against new versions of the compiler to try and catch regressions earlier

[confused-Techie]

So I do want to mention, we have already made stride to push this off onto package maintainers. But it is something that we have not added to our documentation when it really should be there (as well as possibly renaming the repo).

But pulsar-edit/action-pulsar-dependency does exist to let package authors test their package with the latest rolling releases of Pulsar.

Otherwise, I feel it might be rather difficult to find a list of package's to properly test, since being able to test any large majority of our over 11,000 package's would quickly start hitting GitHub API limits, as well as raising costs on hosting the Pulsar Package Repository.

The last concern I want to say, is to be completely honest, we haven't made many if any breaking API changes. The vast majority of breaking API changes have come from only upgrading NodeJS and Electron, which isn't really something we can just say no too.

But of course this would still be very valuable to have, and very valuable to test. So if we are able to find a system that addresses these major concerns then I'd be all for this. Where really this may look like us creating a package ourselves that uses not features of Pulsar itself, but that focuses on only using features of NodeJS, Electron, and the underlying other dependencies we have such as Node-GYP. And with that we'd be able to test these other aspects of the project to get an idea of how community package's break.

[savetheclocktower]

To be honest, I've never really understood how Atom handled this sort of thing. Presumably, whenever they upgraded their version of Electron, they didn't put the onus on package authors to rebuild for the newer version of electron. So I don't see how this is any different. I suspect we could solve this altogether, but I don't have the experience with Electron itself to know how.

[gnmerritt]

Ok well sounds like this isn't a terrible idea so I'll take a stab at getting something working locally before worrying about how it might move up into the cloud. I'm going to start with packages that I personally use + the top 50 or 100 most popular packages.

[confused-Techie]

@gnmerritt Feel free, but to clarify on what I was saying as concerns, the issue of raising costs of hosting the backend, and hitting GitHub API Rate Limits will still affect you doing this locally.

As every package you install for automated tests will hit the backend to install the package, which the backend will then redirect to GitHub to install the final tarball. So if you do work on this it's probably a better idea to start with a lower count of packages, rather than something super high, as well as ensure to implement caching of the packages so we can avoid skyrocketing costs of the backend hosting.

[gnmerritt]

@confused-Techie Ahh that's helpful information, thank you. I will certainly start with a small number of packages and will make sure to not grab them more than once.

[mauricioszabo]

@savetheclocktower as someone that made packages for Atom pre 1.0, my experience is that, yes, they did put the onus on the package authors.

I believe after 1.2 that was when Atom indeed had a "stable" API, but before they I had breaking changes even when without a deprecation warning...