Supply chain security with mise and asdf plugins

[jdx]

mise, like asdf before it, had a major problem regarding supply chain security. This is now a solved problem in mise and I think it's probably the top reason to consider switching to mise from asdf. If you're still an asdf user, you should be concerned about this.

I'll explain this problem and the history of how we got there, but for those familiar with how mise works the short version is this:

Less than 25% of mise tools now default to using the insecure asdf backend. Of those, all of them have been forked to the mise-plugins github org which only I and the mise advisory board have commit access to. Therefore, the supply chain problems that asdf plugins have are no longer a concern for mise users using the default mise registry.

The issue with asdf plugins

asdf tools rely on plugins which contain bash code that is trivial to exploit. (Here's asdf-nodejs as an example). A malicious actor has essentially free access to your box once they can execute bash on it.

asdf plugins also are rarely written by the tool vendor. Generally, plugins are written by people wanting to integrate a specific tool, which then would then get added to the asdf registry, and often abandoned afterwards. Commonly, pull requests to these plugins would never be addressed because the original plugin author had moved on, isn't using the tool anymore, isn't using asdf/mise anymore, or plenty of other reasons.

This meant that it's not uncommon to see asdf maintainers merging PRs that change the default plugin to someone else. asdf maintainers have no good way to vet people making those requests so basically just need to trust anyone making a seemingly correct fix on an unmaintained plugin. If that developer had malicious intent, there are no guardrails in the asdf model to prevent them from later shipping malicious code once they establish ownership.

I want to be clear though: nobody is at fault for any of this. This is open source. Plugin authors have no responsibility to maintain a plugin. asdf authors similarly can't be expected to vouch for people. This also isn't a "bug" in asdf, it's a problem they have not had the time to address and aren't under any obligation to.

Some mise history

Adopting asdf plugins for mise (named rtx at the time), was probably the best reason for mise's success. Instead of trying to build up a corpus of tools to use with mise, mise was instantly useful by relying on hundreds of asdf plugins that were already battle tested over years.

Obviously though, this meant I had the same baggage as asdf and this supply chain issue was going to be a problem for mise as well. It was in the back of my mind but I also had no idea how to fix it.

My first thought was to create the mise-plugins org and fork every asdf plugin into it. At the time (this is around a year ago), I ultimately decided against it with the advisory board as we felt it would be too much work to maintain the full set of asdf plugins. (Note, asdf has a similar org called asdf-community but for maintenance burden reasons gives plugin authors commit access, so it's not really any more secure than plugins by random developers).

I was left without a solution, but a seemingly unrelated feature request I was also working on would eventually get me a solution.

Multiple backends

Meanwhile, some users had asked me about a feature request for making it possible to install simple binary tools from GitHub without an asdf plugin, I believe this was the first but I recall this coming up from multiple people. It seemed like a great idea to me, though challenging to implement.

I took this idea a step further, rather than simply having 1 new way of installing tools, I wanted to have many backends for mise so users could also use package managers like pipx, npm, or cargo to install tools. I'm not sure if I actually connected the dots at the time between multiple backends and the supply chain problem, I was more interested in making it easy to use mise with more tools. Regardless, that would come obviously apparent eventually.

After ~1.5 years, we got there, but there were countless refactors (and corresponding regressions) that were required to make it happen. Even as recently as November I was still doing major refactors for multiple backends to be supported in different parts of the codebase. This was the classic "rebuilding the plane in mid-air" problem where the codebase was originally tied to asdf but needed to be abstracted to other backends. In particular, the concepts of "plugins" and "tools" was initially one-in-the-same. Of course, because most backends don't require plugins this meant I needed to have 2 separate concepts. This wasn't just challenging in the codebase, but also in docs and my messaging. You may still catch me referring to "core tools" as "core plugins" but this is a misnomer I'm trying to get away from (because they're by definition not "plugins").

I don't think I can overstate just how essential multiple backends has become for mise. Just about any CLI can be installed with mise usually without any effort from the vendor. Adding new backends to mise has become pretty easy too, so when users asked for support for things like spm or dotnet, these have only needed about a half day's effort.

Users often mistake mise as "asdf in rust" but that's not at all how I see it. The tagline is "The front-end to your dev env." and an important element of that has been abstracting how tools are installed and switched between versions away from both the user and the vendor. In other words, mise is quite good at managing CLIs whether they're distributed via npm or as precompiled binaries in GitHub Actions. Users can just run mise use -g ripgrep to get ripgrep and don't even need to think about how that's done they can trust that mise will use the most reliable and secure means of fetching it regardless of whether they're on Linux or Windows.

The support for multiple backends only was a prerequisite for the supply chain problem though. In order to actually start migrating off of asdf plugins, we also needed backends to move to. The 2 most important by far are ubi and aqua.

ubi

ubi is a simple rust cli that fetches single-binary tools from GitHub Releases. With ubi, simple CLIs can be fetched directly from vendors without an asdf plugin. First, @autarch did some work to make ubi work as a library for use in mise as well as the integration to use it as a new backend in mise. With ubi in place, we began changing the default backend from asdf to ubi for handfuls of tools at a time to get people verifying that it was working well.

ubi can be thought of as the "zero-config" backend. So long as the vendor has GitHub Releases that are relatively standard, we can fetch tools with ubi with no plugin or manifest.

aqua

aqua is also a tool manager written by @suzuki-shunsuke. If I tried to offer a very rough comparison between aqua and mise's philosophy, I would say that mise puts DX as the primary goal where aqua puts security and reproducibility first.

(as an aside, I love seeing polyglot tool managers like aqua come about. I know for users it can be exhausting with the pace of change in software, but tools like mise only exist because of the efforts of projects before it like asdf, aqua, ubi, etc. Someone will eventually take the work done with mise and build an even better tool which I'm very much looking forward to seeing!)

aqua has a registry that defines all of its tools as yaml files. Typically, these map to GitHub Releases. Here's the ripgrep one for example. This declarative format is great for most of mise's tools since most vendors publish binaries in GitHub Releases. They're much easier to write and maintain than asdf plugins. They also can support extra security features like slsa/cosign though the number of tools that leverage that is low, mostly because vendors rarely offer them.

aqua is written in Go, which means it couldn't be converted into a rust library like we were able to do with ubi. I did not want to require users to manually install aqua separately as I saw it as too core to the DX of mise being used likely for most tools. For that reason, I chose to reimplement aqua's functionality and directly using the aqua-registry yaml files.

This wasn't easy. Among the work to understand and work with the registry files and get mise to behave identically to aqua, aqua also uses 2 different languages that I did not find rust equivalents of. Parsers are not my favorite thing to build (they're hard), but I was pretty committed to the DX here. The first was the go stdlib text/template, and the second was expr. The second one I'm actually a bit excited about because it's a great expression language that I think could be useful in a lot of rust projects.

It's been a bit of a challenge staying up-to-date with the aqua registry. While all tools in the mise registry seem to work great now using aqua, aqua has occasionally added functionality or used a part of text/template or expr that I have not yet built. @suzuki-shunsuke has been extremely helpful working with me and the mise community which has really helped. I'm certain we will have bugs from time to time as we continue to refine our respective implementations, so be aware that may happen but we both work pretty quickly to address them.

Unfortunately one downside of this (which I addressed here) has been that people have been posting issues on the aqua-registry repository that are issues with mise's integration of aqua and not a problem with the registry. I implore you, do not do this. @suzuki-shunsuke is not at all responsible for mise's aqua implementation. I understand the components of mise may not be obvious to everyone, but it's extremely inconsiderate of us to post issues about mise problems on his board. If you suspect an issue is from the aqua registry, make sure you first actually test it in aqua to verify it's not a mise problem. Ultimately, we want @suzuki-shunsuke making aqua/aqua-registry better and not wasting time on problems he can't help with anyhow.

Migration

Starting around fall of 2024, when multiple backends were in place we could finally start mass-migrating tools. Several people contributed PRs migrating hundreds of tools over. I built scripts to try to automate this process best I could. Finally, I put together a checklist of all the remaining asdf-default tools and went through them one by one. If they worked in ubi/aqua, I converted them. If the tool was abandoned, I removed it. That was incredibly painful, tedious work that I never want to do again.

Finally, we got to where we are today where 210/846 (just under 25%) of tools in the mise registry default to the asdf backend.

I've also quit accepting PRs to add any new asdf plugins. If people submit them I tell them they need to use ubi/aqua instead. I'm certain a new tool will come up at some point that can't be done without an asdf plugin. When that happens, it will need to be a very popular tool with clearly no way for us to add it any other way for me to accept it.

Forking to mise-plugins

Now that there were far fewer asdf plugins that mise relied on, I felt comfortable taking ownership of them by forking them to the mise-plugins org where the advisory board and myself will be able to take the burden of owning them. I say "owning" because we won't be able to truly "maintain" them, the community will need to help us with that, but we can at least review/merge PRs. Among other things, the issue of plugin owners not responding to PRs will be a thing of the past for mise users now, at least for tools in the default registry.

Future work

While we no longer rely on any asdf plugins owned by external parties, the supply chain in mise will always be able to be improved.

First, asdf plugins are still a problem even if they're all hosted in mise-plugins. Ideally, the code mise relies on should fit into a single mise version. It's important to have reproducible builds with how users use mise which isn't possible when the plugins are fetched separately.

If you notice a tool you have relies on asdf plugins (you can see which ones by running mise plugins), see if it can be migrated to ubi or aqua. If it has GitHub releases it should be possible, though in some cases it would require some work from either the vendor, ubi, aqua, or mise's implementation of said tools in order to work correctly.

I would also encourage everyone to reach out to tool vendors to see what it would take to offer precompiled binaries for tools that don't, or making similar changes to better support their tools in mise.

I do not think it will be possible to convert every asdf plugin into mise. Some tools require compilation or features like setting env vars which other backends backends cannot do. One solution would be to make all of the tools core plugins, but at least right now I don't see that ever going on the roadmap. It's too much initial work and too much maintenance. mise is also already a hefty codebase and I would be concerned about dumping this much more rust into it.

One solution I've considered is baking all of the asdf plugin bash code for the default registry into mise as assets. That would address the reproducibility problem.

Improving aqua-registry

aqua is the best registry for us to use. It offers extra supply chain security features like slsa-verification and cosign. I intend to make it more visible to mise users when these are being used as a way to encourage their adoption. You can help too by checking if vendors offer these features and if so, adding them to the aqua registry. Also reach out to vendors to see if they'd be willing to build support for those that don't have it.

Other backends?

I would not be in favor of adding any new "general purpose" backends to mise right now (like asdf, vfox, ubi, and aqua). There are reasons to have each of them:

• asdf - large corpus of existing work, easier to author than vfox, supports complex tools not possible with ubi/aqua
• vfox - similar to asdf but a bit more challenging to implement, has some minor perf benefits, supports windows
• ubi - makes it basically trivial to adopt tools that work with it
• aqua - basically the middle ground of everything. Flexible enough that it can support most of our tools, easy to author, supports windows, supports advanced supply chain features. Doesn't work for all tools though (but perhaps we could work to change that one day?)

As the mise ecosystem advances, rather than adopting a shiny new backend, it would be better for us to improve our existing backends, especially ubi and aqua.

vfox?

For simplicity I haven't mentioned vfox which is another backend similar to asdf. vfox is not used by many tools but is helpful because plugins are written in lua instead of bash so they work on Windows. vfox plugins have the same supply chain problems as asdf, but all vfox plugins in the default registry have also been forked to mise-plugins. For the purpose of this article, vfox and asdf are equivalent.

Kudos

Countless people helped make this effort come to fruition. Some like @autarch and @suzuki-shunsuke helped build the lower-level functionality mise relies on, others helped me design many portions of this, and of course mise users helped a lot as well. Some reported bugs but many of those that didn't still had to deal with regressions and bugs that arose from the work required.

I also want to mention @benberryallwood, @scop, @yhakbar, @BurnerWah, @glasser, @jimeh, @yodatak (and possibly others I can't find right now) who all helped contribute to migrating tools off of asdf plugins.

None of this goes unnoticed and I thank you all very much. As most mise users rely only on tools in the default registry, even those unaware of this problem still benefit greatly by having a much more secure setup.

[autarch]

FWIW, I've considered adding support for signature verification to ubi as well. I wrote a bit about this in houseabsolute/ubi#84

That said, from what I can tell, cosign (and signing in general) has fairly low adoption. That's probably because most of the tools folks use to release projects don't support it. I think the best way to get more things signed would be to identify the most popular release tools and help add functionality for this there. Then it's a lot easier to file an issue for a project asking them to adopt this, as opposed to asking them to do all the work of signing from scratch.

[jdx]

That said, from what I can tell, cosign (and signing in general) has fairly low adoption.

This is one of the reasons I wrote this. I see this as the initial foray into educating users about how their tools are fetched and why they should care. I think this is something we can all help with and there is a lot that can be done. Tools like mise/ubi can start to leverage them more and make it clear when they're being used (and maybe when they are not). I agree the tooling also needs to improve so it's easier for vendors to adopt.

Notably I also added a lockfile to mise which does verify checksums which works with the ubi backend. Of course, this isn't gpg verified or anything like that, but at least users have a way to ensure that they are all getting the same exact binary in a particular project.

[autarch]

Yeah, there's probably a bit of a chicken and egg problem here too. Why would a vendor add signing if users haven't adopted tools which verify signatures. So adding support for this to things like mise and ubi would help justify the work that vendors need to do in order to add signing.

[glasser]

This is really well written and helpful to understand the full story here.

One question: let's say we are using a tool that still uses a well-maintained asdf plugin and want to contribute a fix or improvement. Would you prefer mise users start with a PR to the mise fork, or with a PR to the upstream plugin followed by a request to port it over? (For any plugins where upstream gets poorly maintained, I assume the former is fine.)

[jdx]

Good question. We've been in this state already since even before the forkening there were lots of forked repos in mise-plugins. People have been making PRs to both. Obviously, that's not ideal but I'm not sure what we could do better. Alternatively, people could make an upstream PR and send me a ticket/message asking to sync the fork. The latter might involve slightly less work for everyone.

The good thing is plugins rarely need changes so this shouldn't be something people need to do very often.

[pepicrft]

Great write-up @jdx and amazing work to everyone that got involved in making this happen.

Ideas take time to emerge and click, and this is a great example of that. Mise started with DX and found itself getting the ecosystem to a much safer place, which benefits many projects out there.

Can't wait to see what the future holds for Mise.

[reitzig]

Seconded.

I ended up with using both aqua (for its registry) and mise (for its DX) and now it seems I may be able to drop aqua the tool at some point. Renovate support isn't quite there yet, but that's only a matter of time.