Modules: Node/Browser ESM Interoperability

[giltayar]

How does this group see ESM interoperability between Node and the browser?

(In all the discussion in this issue, we should assume that the Node/JS code in question can work in both environments)

1. Will we be able to take current browser ESM code and have it work in Node, transparently (i.e. without a build step that generates a distribution that is fit to run in Node)?

2. Will we be able to take any Node code that uses only ESM, doesn't have bare imports, and always imports using relative paths _and_ a file extension, and be able to run it transparently in the browser (i.e. used with a script src (type=module), but without a build step that generates a distribution that is fit to run in the browser)?

3. Will we be able to take Node code like the one in question 2, except that it imports without using a file extension, and have it run in the browser using script src(type=module)? Given that currently the answer is no, and assuming we want to answer yes, what is the mechanism by which this will work? Is it a build step, a service worker, a smart http server that serves the file, do we work with the TC39/browser vendors to enable this, or is there another option?

4. Will we be able to take Node code like the one in question 2, except that it uses bare imports (resolved using the regular Node ESM module resolution algorithm), and have it run in the browser using script src(type=module)? Given that currently the answer is no, and assuming we want to answer yes, what is the mechanism by which this will work? Is it a build step, a service worker, a smart http server that serves the file, do we work with the TC39/browser vendors to enable this, or is there another option?

5. Will we be able to take an "npm app" (my terminology for an app that has a package.json with the required dependencies) that only uses ESM, be able to npm install it, and have it run in the browser by script src-ing the entry point? Given that currently the answer is no, and assuming we want to answer yes, what is the mechanism by which this will work? Is it a build step, a service worker, a smart http server that serves the file, do we work with the TC39/browser vendors to enable this, or is there another option?

(The first two questions are pretty trivial, as the answer _today_ to both of them is yes, but I'm putting them here as a baseline and to see whether I missed something. The other questions are... more difficult.)

[giltayar]

Sorry about the legalese, but everybody seems to have their own notion of what this means, so I wanted to be as clear as possible what this issue is about.

[bmeck]

@giltayar I think the links got mixed up.

I'd like to link to the WHATWG hooks issue: https://github.com/whatwg/html/issues/2640 , but I haven't been able to gather browser implementer interest in such a thing. They want to wait on userland.

We should also probably bring up MIME since if/when we ever support data: URLs we have to have it match the browser, and that browsers intend to use MIME to disambiguate dependency format.

[WebReflection]

My take

Considering how ESM already concretely shipped everywhere else, which is behind a flag that opts in for ESM, NodeJS should be _at least_ able to run every browser code, that uses compatible ECMAScript primitives or classes, without any issue.

basic scenario (covers 1 and 4: Yes)

The following scenario is the minimum requirement, where both _relative_, _absolute_, or _fully qualified URLs_ should work.

a.js

import b from './b.js';
console.log(b);

b.js

export default Math.random();

The execution of node -m a.js, where -m could have any meaningful name (--type=module to mimic browsers, --esm, others) should output a random number in console.

jsc -m a.js already works (using print instead of console.log), and so does js52 -m a.js.

Any browser with a <script type=module src=a.js></script> would work too.

100% ESM portability 🎉

WHATWG Hooks (Covers 2 and 3: Yes, we might be able but ...)

Whenever WHATWG will make a decision about hooks, NodeJS should adopt a similar solution.

However, hooks should be non blocking for the shipping of ESM or we'll be stuck in a chicken/egg issue.

Irrelevant Extras

AFAIK there is no use case to import data: URL in NodeJS. Unless proven differently, I don't think we should ever care and agree that data URL is non blocking for ESM.

npm compatibility

Once ESM works in NodeJS, authors of modules can already ship portable code that works in both browsers and NodeJS (and vice-versa, as long as there's no core module involved).

core/npm modules (Covers 5: Yes anyway ...)

Hooks might solve the issues for the browser land so that NodeJS might never need to even implement hooks since hooks are meant to solve the module loader resolution, which is already working in Node (the CJS part that needs revision to work with ESM packages too)

However, it is already possible to use unpkg.com cdn to ask pure ESM for browsers and resolve module paths.

Example, visit https://unpkg.com/hyperhtml-element?module and see that the first import resolves to a fully qualified URL compatible with ESM instead of what the source file would show (which would require tooling otherwise).

That means the ecosystem around ESM alredy made npm packages work for ESM and browsers.

[bmeck]

I'm fine dropping data: but we still need to ensure that our format negotiation scenario meshes well for any loading mechanism and be sure we don't introduce anything that conflicts with browsers using MIME to disambiguate dependency formats. I don't think we can remove MIME entirely from the interoperability story.

[WebReflection]

be sure we don't introduce anything that conflicts with browsers

not implementing means we won't possibly conflict with anything. It won't be there but it could be eventually implemented.

[bmeck]

not implementing means we won't possibly conflict with anything. It'll be just there ready to be, eventually, implemented.

I'm not sure I understand the point of this.

[giltayar]

@bmeck

We should also probably bring up MIME since if/when we ever support data: URLs we have to have it match the browser, and that browsers intend to use MIME to disambiguate dependency format.

Could you explain the issue (or link to something that explains it), and I'll add it to the list of questions?

[giltayar]

@WebReflection - In the interest of focus, I'd like to focus on the questions I defined in the issue. I'd love it if you could rephrase what you wrote (on your take) as answers to the above questions? And If what you wrote is beyond the scope of these questions, I would appreciate it if you could open other issues on it.

[WebReflection]

I'm not sure I understand the point of this.

browsers and other envs distinguish between JavaScript and everything else. Everything that is JavaScript works as ESM when you use the module flag.

data:, whenever it will be needed, would use a JavaScript mime type and it will work as ESM.

All this answer the question: what does it mean to have ESM compatible/interoperable with browsers.
There is no MIME type issue, with browsers.

[WebReflection]

@giltayar I've described what it means ESM interoperability with browsers. I'll update the post with straight Yes/No around examples/points.

[inidaname]

not implementing means we won't possibly conflict with anything. It'll be just there ready to be, eventually, implemented.

if eventually implemented then it still goes back to @bmeck's

be sure we don't introduce anything that conflicts with browsers

why add a feature that may never be use, it makes no sense investing in something you sure won't make it to production and you are sure to conflict whenever it eventually make it to production.

[bmeck]

@giltayar

By choosing to match browsers in using URLs to back our ESM Module Map / specifiers we have some implications about using MIMEs.

First, data: URLs contain their own MIME signature per the standard in RFC 2397. Since we explicitly don't allow these URLs currently, we can leave it for future consideration and not consider it blocking.

In addition, the second clause is about the fact that browsers are using MIME as a means of disambiguating and determining the format of dependencies:

• WASM
• HTML Modules
• JS / WHATWG type=module

We just need to be sure that whatever disambiguation scheme we use can match and scale in the same way. In particular with the use of Node specific MIMEs we should just be clear we can't cause collisions.

[WebReflection]

@inidaname in case it wasn't clear, I am for not implementing stuff that is unrelated to NodeJS

[WebReflection]

@giltayar I've updated my take covering all points and adding extra examples/solutions

[bmeck]

@WebReflection I'm fine not implementing MIME for any first iteration, but we do need to regard that as being the means of disambiguation being used by browsers/servers. Any disambiguation mechanism we use needs to be capable of mirroring MIME's capabilities. At which point I generally would say we should just use MIME. Use of data: contains MIME by nature and any introduction of canonical URLs like blob: from URL.createObjectURL would also be using MIME.

[WebReflection]

@bmek as long as MIME does not force a distinction between js and js I have zero objections. Actually, browsers won't work without a mime type of JavaScript, in case the file is served as plain text.
Yet I don't think we should cover mime here or now, since CJS just worked with extensions 'till now pretty well.

[bmeck]

@WebReflection MIME does contain conflict for .js meaning both application/node and text/javascript.

[WebReflection]

--type=module syntax.js should handle JavaScript syntax as ESM. If application/node would do that, I'm OK. If it doesn't MIME can wait since it's a non blocking issue for NodeJS that always worked with just extensions.

[bmeck]

@WebReflection that flag based approach is an entirely different disambiguation scheme than MIME is intended to be used as. We would need to do various ways of putting that into MIME such as https://github.com/nodejs/node/pull/18392 and/or https://gist.github.com/bmeck/7ee7eb2147e2dafe3167c856d9b4151a

[WebReflection]

then again this whole thread is about the flag. All questions are about the flag. I think MIME can stay away from this team for the time being. I think we are here to accelerate and solve issues, not to preemptively bring issues we don't have. MIME is non blocking, let's move forward ?

[bmeck]

@WebReflection I would block on MIME compatibility since that is what browsers are planning to use.

[WebReflection]

you wrote this:

I'm fine not implementing MIME for any first iteration

what's the point in blocking what? we don't have any incompatibility with browsers here

[bmeck]

My block would be that I consider

Any disambiguation mechanism we use needs to be capable of mirroring MIME's capabilities

to be a requirement for any intention of browser interoperability.

[WebReflection]

there is already interoperaability with browsers in terms of ESM and JavaScript. What other problem do we need to solve? What is the problem you are talking about? Any example?

[bmeck]

The problem most easily seen with files that have MIME collisions, notably .js has 2 meanings in Node. We just need to be able to explain and disambiguate those MIME using some mechanism. In addition there are more out of band data bits coming to JS. Notably the JS MIME is gaining the goal= MIME parameter so it should scale against any parsing modes being added to ECMA262 and https://github.com/domenic/proposal-function-prototype-tostring-censorship was discussed last TC39 as also wanting another out of band mechanism which I presume will become another MIME parameter. These are just things to prepare for, not things that need to ship.

@bmeck can you please remove death proposals/standards such HTML Modules from this list?

Death?

[evanplaice]

1. Yes, but for relative path imports only. Bare imports are not supported.

2. Yes, if it's ESM and relies on only relative imports it should just work. I have experimented with this lately and it works very nicely.

I experimented a bit with assetize (the npm asset proof-of-concept) project and it worked very well with ESM imports. I could see wider adoption of this approach by FE ecosystem for packages that either don't contain JS or those that shouldn't expect to rely on NPM (ex FontAwesome, Bootstrap, etc). Keep in mind, there are a lot of package managers out there (ex composer) that ship FE dist packages with no knowledge of or support for NPM.

3. No, as you've stated. No extension, implies that the import is a bare import. It will be relative to the package root.

For example:

import { observable } from 'rxjs/dist/rxjs.umd'; // note missing .js extension

Will load the source from ./node_modules/rxjs/dist/rxjs.umd.js. This is how transpilers currently resolve bare imports.

Source: I have used this approach to include pre-transpiled packages with both System.js and Webpack.

4. Currently won't work. ESM in the browser currently only supports relative paths. Bare import support would need to be addressed by WHATWG.

5. See 4.

---

As far as bare import support in browsers goes. It would be ideal if they specified an element like <base> that specifies the root path for modules to resolve to. But, we're not WHATWG so that falls outside the scope of this group.

[bmeck]

@WebReflection HTML Modules are a different proposal from HTML Imports.

[WebReflection]

HTML Modules are a different proposal from HTML Imports.

apologies I've misread that. Then if you talk about script type=module yes, that's what I'd love to see in Node too.

[tbranyen]

WHATWG Hooks (Covers 2 and 3: Yes, we might be able but ...)
Whenever WHATWG will make a decision about hooks, NodeJS should adopt a similar solution.

Just posing a question related to this discussion. Should Node wait on WHATWG? Seems like a first-vendor-implementation problem that could just as easily be spearheaded by Node and followed by browser vendors (if it's sufficient). jQuery has influenced browser API design, I don't see why Node should be limited by browser concerns when we could design something that bridges the divide as a primary use case.

[bmeck]

@tbranyen the existing Loader hooks should be compatible with that if it ever makes progress. You can do pretty much everything using just a resolve hook when you can generate ESM in memory / reserve URLs. The tricky part is making that API nice to use. I wouldn't block us on adding new things, but for the most part we can get more out of the resolve hook than other additions. Still need to ship a good URL reservation mechanism though (if someone wants to PR URL.createObjectURL that is one way to reserve URLs with content).

[devsnek]

for our members that don't speak chinese https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fcnodejs.org%2Ftopic%2F5c3f1323a4d44449266b1be4

[MylesBorins]

Closing this as there has been no movement in a while, please feel free to re-open or ask me to do so if you are unable to.