Berry: 🎯 ESM Support

[zdm]

What package is covered by this investigations?```
@softvisio/cli

Describe the goal of the investigation

I am trying to run script, that uses es6 syntax.

yarn add @softvisio/cli
yarn softvisio-cli

But get this error:

(node:4268) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
d:\downloads\yarn\@softvisio-cli-npm-0.1.0-efe3491d08-1.zip\node_modules\@softvisio\cli\bin\cli.js:3
import softvisioCliApi from "../lib/api.js";
^^^^^^

I am using node 13 that can load es6 modules directly.

Is it currently possible to use pure es6 modules with yarn 2 or I need to compile them to commonjs before?

[arcanis]

ESM modules go through a completely different codepath in Node (cjs / esm). The problem is that this codepath is still very new, and although modules have technically been marked stable the way to alter the resolution is still in heavy discussion (https://github.com/nodejs/modules/issues/351).

So we can't support them at the moment (too early), but as soon as the loader spec moves forward with an implementation we'll be ready to experiment with them 🧪

[zdm]

Thank you.
Please, don't close this issue until this feature will be implemented.

[bgotink]

I looked into the API a bit using Node 14. I've written a PoC implementation at (https://github.com/bgotink/berry/commit/34c0a7b19fbd16191f558b80e3ad296dbb7d7508) along with a small test repo over at https://github.com/bgotink/berry-esm.

The API is still unstable, but I think it's worth checking out to see whether there's any valuable feedback to gain from trying. I'm not sure how volatile we should consider the API, so I'm not considering "should we ship this?" at the moment, only "does it work?"

The interesting bits so far:

• It works.
• Resolving a CommonJS dependency using {format: 'commonjs'} in the getFormat hook gives us no more hooks into the file's resolution, so we can't load it from the yarn cache's zipfiles. The same goes for JSON files and {format: 'json'}. All of these need to go through {format: 'dynamic'} which is kinda sad because that means we need to replicate the Node behaviour w.r.t. named exports in these kinds of files.
• This will go nicely with support for package exports (https://github.com/yarnpkg/berry/issues/650), where the regular .pnp.c?js file asks for CJS files and the ESM loader uses import. In fact, I would assume most packages that want to adopt ESM will do this in a backwards compatible way, in which case we need package exports to load the ESM variant of the library.

The boring bits:

• Passing the --experimental-loader flag logs a warning regardless of whether said loader is actually ever used. In other words, so to not harass users not interested in experimental ESM support we should make this opt-in
• This is probably not compatible with the native ESM loader. For instance, the loader powered by PnP is quite lenient w.r.t. file extensions, while the native ESM loader isn't.
• Some things don't work yet, e.g. import(await import.meta.resolve('typescript')) fails at the moment because the ?yarn-cache flag gets lost in the resolver
• Stuff like import(pathToFileURL(createRequire(import.meta.url).resolve('typescript'))) doesn't work.
• I've explicitly kept the pnp-loader static and use the PnP API exposed from the .pnp.c?js file

Improvements to the PnP API that would make the PoC a lot cleaner:

• Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)
• It would be great if pnp could be given an option to make file extensions required when directly referencing a file
• A function to see whether a given file path is part of the yarn cache or not would come in handy, that would allow us to use regular file: URLs instead of using a ?yarn-cache query parameter.

A node module with code below works fine in the berry root folder and showcases a lot of the basic functionality already.

// named exports from fs still work even though we patch fs
import fs, {readFileSync} from 'fs';

// JSON can be imported
import pkg                from 'typescript/package';
// CJS package import
import typescript         from 'typescript';

// importing a non-patched builtin
import {fileURLToPath}    from 'url';

console.log('');
console.log(pkg.name, typescript.version);

// Resolves to a regular file: URL
const typescriptUrl = await import.meta.resolve('typescript');
// So it can get transformed into a path and used in fs APIs or even require (BAD IDEA)
const typescriptPath = fileURLToPath(typescriptUrl);

console.log(`url:`, typescriptUrl);
console.log(`path:`, typescriptPath);

const eslintPackagePath = fileURLToPath(await import.meta.resolve('eslint/package'));

// Just to prove that the fs APIs are patched
console.log('fs.readFileSync', JSON.parse(fs.readFileSync(eslintPackagePath)).version);
console.log('readFileSync', JSON.parse(readFileSync(eslintPackagePath)).version);

$ node --experimental-loader ./.pnp-loader.mjs --experimental-import-meta-resolve --harmony-top-level-await test.mjs
(node:68602) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

typescript 3.8.3
url: file:///Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js?yarn-cache=
path: /Users/bram/Workspaces/Private/berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript/lib/typescript.js
fs.readFileSync 6.8.0
readFileSync 6.8.0

[arcanis]

Nice analysis! You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

Ideally we'd store the package's type in the PackageInformation exposed by PnP, instead of having to read it from the manifest file ourselves. (The same will go for package exports.)

That seems acceptable 👍

It would be great if pnp could be given an option to make file extensions required when directly referencing a file

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

[bgotink]

You mentioned ?yarn-cache - why is this query parameter required? Regular paths currently seem to work fine 🤔

I've kept the module loader conservative in which paths it actually tries to handle itself versus which paths it lets the default loader handle. This requires the different hooks to be able to check whether a URL points to a path that needs to be loaded via the patched filesystem.
The ?yarn-cache query param was a quick and dirty solution for that. Another possibility is an API to see whether a given path requires loading via our patched fs module. Alternatively we could just load all files via the patched fs, though that makes the loader anything but conservative.
In a first attempt for the loader I used a custom protocol yarn-cache:, but it's a bad idea to stray away from the file: protocol for compatibility reasons: import.meta.resolve(...) and import.meta.url expose the URL of a module, changing that from a file: URL to something else will cause breakage for people using the module's URL in order to e.g. load a data file in the same folder as the code file.

I'm not sure what's the best path forward here: just load everything via our getSource hook or defer to the native hook when possible? (in the latter case we need a mechanism to devise whether a URL should be loaded by us, beit via the current ?yarn-cache query parameter or via an API call)

Calling resolveToUnqualified might have the same effect as such a flag. Worth a try.

resolveToUnqualified doesn't resolve bare imports to their actual files, it resolves typescript to /.../berry/.yarn/cache/typescript-patch-c42e3ab680-2.zip/node_modules/typescript.
At the moment the PoC loader uses resolveQualified to resolve that to an actual file, but that implements too much: it automatically adds file extensions and it resolves folders to files called index.whatever.

As I've stated in my previous comment I've kept the module loader as dumb as possible, it loads everything from the regular .pnp.c?js file. Implementing both the CJS resolution algorithm and the ESM resolution algorithm in the regular pnp file is probably not the best idea. Maybe the solution is to use resolveToUnqualified and then handle the rest of the resolution in the module loader?

[dezren39]

What is the timeline for enabling ES support? Is there anything one can do to work around this? The issues linked to this suggest not. For now, I will try using berry with the nodeLinker option and see if that lets me use berry, though with it's biggest feature limited lol. The reason I want pnp is essentially what you've labeled 'zero installs', before this I used yarn 1 and the offline mirror mode to make exactly the same .zip folders that berry makes, even in the .yarn folder and everything lol allowing 1 zip per dependency and sane commit changes by just checking the .zips. Built it all with docker images lol. So I'd hoped berry would be able to work in the same way, but since there actually isn't ever a node_modules with this method, I see how it will be more difficult to patch, because the offline-mirror was just how it got stored in git, but once running it was identical to npm/node with a node_modules folder.

[desmap]

@arcanis @bgotink any news here? @arcanis left the last comment on the related thread https://github.com/nodejs/modules/issues/351 on on Jul 29, 2019, so is there some hope?

[arcanis]

Of course there's hope, and it's even certain it'll eventually happen. It's however not my top priority at the moment since all the software I personally use are still CommonJS, so it'll happen faster if someone can do the legwork 🙂

[desmap]

I personally use are still CommonJS

Most of the stuff I use is also CommonJS, I agree. However, once you start to code share between frontend and backend and already bigger libs like Next default to esnext modules it's just cleaner and less headache to have all the code base as one format.

Currently, code sharing is a rocket science. But that's not your fault ;)

[desmap]

or another example which just popped up again in my project: top-level awaits, they are such a life saver and way underrated. Once you get the hang on them you can do insane stuff. Right now I try to setup conditional imports (which are of course async) and then continue depending on the condition. With top-level it would be DRY, without, a lot of .thens, code dupes and just messy.

But TS requires esnext modules for using top-level await... so yeah

[nnmrts]

since all the software I personally use are still CommonJS

@arcanis Can you please put that ESM limitation of Yarn 2 in the documentation or the "getting started" or the "migration" guide? Maybe that would save others from bloating up their project with zip files thinking a new version of a tool would have more features than the old one, not less.

Node 14 supports ESM, which is the version I'm using. I have type=module, my code is supported in node itself, so I don't get why it shouldn't work in Yarn.

This script/module doesn't import any external module, it's just importing another local ES module. I don't know why this is not working and how this is not documented anywhere. How is this expected behaviour for anyone? Why is Yarn taking over my node and effectively limiting it by taking away features it previously had? Is it using @babel/node now? Or @babel/register? If yes, why?

Welp, whatever, going back to Yarn 1 now, let's see if Yarn 3 in 2021 will support the widely accepted and used standard to write javascript modules, I guess.

[arcanis]

Hey @nnmrts - before commenting in an open source project (any open source project), please remember that the people in front of you aren't your contractors but your peers. Passive aggressiveness like "if you don't do X I won't use Y" isn't more likely to make X happen, because maintainers don't work for you (at least I haven't received the wire yet).

That being said, if you need to use ESM now, you just have to downgrade to typical node_modules installs using the nodeLinker setting and you'll have your work done for you. If you need to use ESM with PnP then it's perfect as we happen to be looking for someone with free time to help us on this. Ping us on Discord and we'll put your energy to good use for the community! 🙂

[merceyz]

I gave this a go and implemented an experimental loader in https://github.com/yarnpkg/berry/pull/2161, if anyone would like to test/try it you can use yarn set version from sources --branch 2161

[SagnikPradhan]

Guess what? Node.js latest release has officially declared ESM API stable. But it seems like WASM, JSON and Custom loaders are still experimental.

References - 1 2

[Bnaya]

Guess what? Node.js latest release has officially declared ESM API stable. But it seems like WASM, JSON and Custom loaders are still experimental.

References - 1 2

esm yes, esm loader hooks, which is the needed part: No.
Most of the machinery of ESM in nodejs that needed for most tooling is still experimental

[SagnikPradhan]

Most of the machinery of ESM in nodejs that needed for most tooling is still experimental

Oof. IG We will have to wait a couple more months.