Allow specifying which packages are included by packs-rails

[TastyPi]

Currently packs-rails loads all packs, but it would be really useful for us if we could specify which packages were loaded. This would enable support for two things:

1. Testing packages only loaded for tests, see this discussion
2. Multiple Rails apps sharing packs

2 is something I've been exploring but haven't fully fleshed out, basically we have a monolithic Rails app that supports multiple web apps and APIs that we would like to split out into separate servers, but with shared code loaded using packs-rails. This isn't really possible if packs-rails always loads all packs.

Would you be open to pull requests implementing this? My thinking for how to implement it would be to have a package.yml in the Rails app's root and use the dependencies defined there as the packs that should be loaded, WDYT?

[alexevanczuk]

Hi @TastyPi thanks for starting the discussion. I spoke a bit with my team about this and we're not quite ready to support this. Having packages dynamically load based on dependencies and other configuration will introduce a lot of complexity that we're not sure we'll be able to support in a satisfactory way (and Gusto wouldn't use this feature, making it hard for us to verify behavior from changeset to changeset).

I have a couple of possible stop-gaps for you for the time being:

• The first would be that if your packages are able to be loaded across apps, that means that the nested dependency tree of those packages is actually quite clean. That is -- they're not all entangled in a giant cyclic dependency web. If this is the case, it should be a bit more straightforward to turn these packages into gems or engines. bundler and Gemfiles have a much more well-supported ecosystem for sharing across applications. One of my hope with the packs ecosystem is that going from pack to gem and back has a nice golden path so that could be a valuable thing to explore.
• Another option that is perhaps not as satisfying (but it seems like you might be doing based on your comments in the slack server) is playing around with your own implementation of loading packages based on packs-rails. It never feels too great to have to work off of a fork but it might be one way to get this without too much extra work and also would provide some time to experiment and explore the approach.

I wish I could give a more desirable answer for you here – let me know what you think.

[TastyPi]

We're nowhere near having a clean dependency tree, the multiple Rails apps is a hypothetical beautiful future that I would like to reach one day after we've finished modularising our monolith. I mostly wanted it now for my suggestion of the test support code, but I like the suggestion you gave in the other discussion.

I think you're right that gems/engines would be a better long-term solution for sharing between multiple Rails apps. It would also solve gem dependency management (e.g. if a pack depends on a gem we might not want that gem to be included in the apps that don't use that pack).

I'll keep playing around, but I am concerned about your comment that if a feature isn't used by Gusto it won't be accepted into this repo. I understand it increases the maintenance burden, but it makes me wonder what the goal of this project is. Is it just intended to only support what Gusto needs and is open-sourced for projects that don't need anything extra, or is it intended to be the "industry standard" way to modularise Rails apps? If the former then fair enough, we may have to continue maintaining our own solution instead, but if the latter then I don't think it's going to succeed unless you're open to features Gusto doesn't need (with the assumption that the projects who do need those features and depend on packs-rails will help with the maintenance burden). I'd like to know this project's position before I spend too much time working on packs-rails instead of just building exactly what we want.

(FYI I actually have implemented adding support for specifying which apps get loaded and it wasn't particularly difficult, you can see the changes needed here rubyatscale/packs@main...Ophelos:packs:transitive_dependencies and here main...Ophelos:packs-rails:config-packs (no tests since it's just a proof-of-concept and possibly no longer needed, but thought I'd share anyway))

[alexevanczuk]

Hey @TastyPi , thank you for this comment and I appreciate the sentiment and pushback. Our goal is definitely not to only support what Gusto needs, but to help the industry improve on its ability to write modularized ruby code. (Not sure if I'd say the goal is to become the industry standard, since I'd be happy if another better solution came along that helped solve folks' problems better.)

I wholly agree that for success we'll need to find some way to support a broader flexibility of use cases. In the nascent, experimental stages of this project, we have to be careful about adding too much functionality that we might not be able to support, since I believe that is another vector for folks to be disillusioned by this project, and depending on other orgs or independent contributors for maintenance is something I hope we can reach as we mature.

That being said, there is a more technically concrete way we can represent the idea of others maintaining their own complexity: the use of injected dependencies and configuration. In this case, that might look like allowing packs-rails to read a per package.yml field like require: false, but having the value be injected via ERB templating by the client who maintains their own complexity on how loading works and its downstream impacts on the rest of the ecosystem.

However, in the interest of supporting the industry, I'm wary of us having too much cross-over with the functionality of gems, the industry de-facto standard for package distribution. One way we could support the industry here is to work on tooling that makes converting a package to a gem easier. Note that "distribution" is currently a non-goal of packages since we believe gems are and will continue to be the ruby standard for distribution.

Thanks again for your feedback, let me know your thoughts.

[TastyPi]

Thanks for the clarification, and I'd like to add my comment on only supporting what Gusto needs wasn't intended to sound like a criticism, it's a completely valid position to take, I just wanted clarity.

I've been tweaking our setup a lot as we learn what's useful to have and what just adds complexity, and I've been keeping in mind packs-rails as a "it would be nice to be as similar as possible if we don't use it". It's now at least built on top of packs, and frankly the only major difference is we use minitest instead of rspec, so the FactoryBot and RSpec integrations don't work for us. Adding minitest support feels like a less controversial feature request, though I'm thinking it might make more sense to have a separate packs-minitest package.

Being able to specify exactly which packages are loaded is no longer needed for how we set up tests, but would still be needed for the long-term goal of multiple Rails apps. The suggestion of a require: false field wouldn't work for that situation since it should be the Rails app not the package that decides whether it gets included. We are however a long way off from being able to split our monolith into multiple Rails apps, I got a bit overexcited with mentioning it now, so I'm happy to table this discussion until it's something we (or someone else) actually needs.

[alexevanczuk]

It's all good – I appreciate the request for clarity and even if it were criticism, I welcome it as feedback on how to possibly have greater positive impact.

A packs-minitest gem sounds great to me – we actually hope long-term to split out packs-rspec and packs-factorybot, but haven't done it yet mostly in order to attempt to throttle public API changes a bit.

With require: false, I really meant something like require: <%= MyLoader.should_load?(__FILE__) %> or something like that, where the Rails app can basically "inject" the loading directive based on application needs. That being said -- we are on the same journey as you here and are also hoping to be able to split up our monolith. The North Star vision (more of a direction than a definite end state) here is that by using the packs ecosystem and gradually modularizing a packs-based rails app, the build and deploy system can automatically decide to deploy the app as a monolith or as independent "micro services" based on the modularity progress. We have a long, long way to go though!!

Thank you again for being a thought partner here 🙏