Implements PnP support

[arcanis]

(note: each commit contains a particular semantic change, so you can review them one by one if you prefer)

This PR implements support for PnP. I opened it in draft since I'm sure some feedback rounds will be required, but it should already be working fine (added some tests to verify that).

There are two main changes:

• During the node_modules resolution we first check whether a PnP manifest is attached to the resolver. If it is we query the PnP resolver to locate the package directory instead of performing the typical node_modules traversal. This directory is then used in oxc's subsequent resolution passes (exports, extensions, etc).

• Since Yarn stores the packages inside a zipped cache (details here), oxc needs to know how to look into them. The pnp crate provides some utilities to make this work simpler, which I used here. It adds some boilerplate, I'm open to ideas to clean it up (but since it's in fairly low amount code in absolute measures, perhaps it's ok?).

There's one known limitation: PnP technically supports interlacing completely different dependency trees. This only happens when a project with a yarn.lock tries to import files from another project with a different yarn.lock, which is very rare (it usually only ever come up when using some specific scenarios of yarn dlx). I didn't implement it here, the manifest is instead passed as option (but perhaps we should instead locate the manifest by crawling the filesystem?).

Fixes #53

[Boshen]

I'll start experimenting with this feature locally by initiating a yarn pnp project.

[Boshen]

Things left for me to do:

• port tests over from enhanced-resolve https://github.com/webpack/enhanced-resolve/blob/main/test/pnp.test.js
• decouple the file system integration so the rest of the tests pass
• add a yarn pnp feature flag (disabled by default)
• add a integration test on CI: install yarn pnp and resolve

[arcanis]

The tests should work, I may have broken something before push; checking it now (my intent is that the presence or not of the PnP manifest is the feature flag - it should have 0 impact on people that aren't using PnP)

[Boshen]

The latest changes is working on my end:

With a new project:

{
  "name": "test-yarn",
  "packageManager": "[email protected]",
  "dependencies": {
    "lodash": "^4.17.21"
  }
}

Update the options in examples/resolver.rs:

    let options = ResolveOptions {
        pnp_manifest: pnp::load_pnp_manifest("/Users/boshen/github/test-yarn/.pnp.cjs").ok(),
        ..ResolveOptions::default()
    };

cargo run --example resolver -- /Users/boshen/github/test-yarn lodash

prints

path: "/Users/boshen/github/test-yarn"
specifier: lodash
Resolved: "/Users/boshen/.yarn/berry/cache/lodash-npm-4.17.21-6382451519-10c0.zip/node_modules/lodash/lodash.js"

[codspeed-hq]

CodSpeed Performance Report

Merging #217 will not alter performance

_{Comparing arcanis:mael/pnp (ca3030c) with main (b9d0443)}

Summary

✅ 3 untouched benchmarks

[Boshen]

my intent is that the presence or not of the PnP manifest is the feature flag - it should have 0 impact on people that aren't using PnP

I understood that the option pnp_manifest is the runtime feature flag.

Unfortunately Cargo will pull in and compile all the unused dependencies. We need to a feature flag to avoid these extra dependencies at compile time.

[Boshen]

Thank you arcanis, this integration is really smooth.

I'll take care of the rest. Will ping you or create an issue in pnp-rs if any problem comes up.

[arcanis]

That's unfortunate :( Would it help if we had fewer dependencies?

My concern is that every tool using oxc-resolver would in theory benefit from having PnP support; if they have to do something to get it to work, they may forget doing it (the necessity of manually passing the pnp_manifest option is already adding this friction, but I was thinking it could be removed by crawling the filesystem to locate the manifest).

[Boshen]

My concern is that every tool using oxc-resolver

The tool will decide whether they want to enable pnp.

All the Rust bundlers will enable pnp. Simple tools that only require simple resolution will not enable it.

the necessity of manually passing the pnp_manifest option is already adding this friction, but I was thinking it could be removed by crawling the filesystem to locate the manifest

I'll take a look at how enhanced-resolve and parcel handles this problem.

[arcanis]

enhanced-resolve and parcel handles this problem.

Enhanced-resolve checks whether the process is running within the PnP runtime, so no configuration is needed. Parcel doesn't support PnP yet (I only prototyped it a year ago, but didn't open the PR).

[Boshen]

Thank you for the context, I have gathered the following information:

• webpack sets pnpapi as a builtin https://github.com/webpack/webpack/blob/dd44b206a9c50f4b4cb4d134e1a0bd0387b159a3/lib/node/NodeTargetPlugin.js#L71
• enhanced-resolve uses require("module").findPnpApi https://github.com/webpack/enhanced-resolve/blob/fd3bde2b8eb18eb2138f3a9cadebcfee26922d6f/lib/ResolverFactory.js#L132C5-L132C33

For bundlers with napi integration, they can get the manifest path in node.js and pass it over.

So the question is, are we able to get the manifest path from require('pnpapi') or similar APIs provided by the runtime?

For tools without a node.js runtime, they'll have to provide the path explicitly as a compromise.

[arcanis]

So the question is, are we able to get the manifest path from require('pnpapi') or similar APIs provided by the runtime?

Yes, there are two ways:

• Calling require.resolve('pnpapi') will return the path to the current .pnp.cjs file
• Calling require('module').findPnpApi?.(cwd) will crawl the filesystem starting from cwd (which can be __dirname, process.cwd(), ...)

[BasixKOR]

It would be worth looking into what esbuild does for detecting PnP manifest since using pnpapi would result in oxc-resolver loading up a Node.js instance every time the Resolver instance initializes.

https://github.com/evanw/esbuild/blob/360d47230813e67d0312ad754cad2b6ee09b151b/internal/resolver/resolver.go#L514

[Boshen]

I'm going to merge this and continue working on the main branch.

Thank you arcanis for the hard work.