Modules: Web Modules for Node

[zenparsing]

Over the past week or so I've been experimenting with a non-transparent interop solution I'm calling Web Modules for Node. The basic idea is that instead of trying to create a single overarching module design that encompasses the strenths of both CommonJS and ES modules, we instead reframe the effort in terms of adding support for "browser" modules, similar to how node has been adding support for other traditionally browser-based features. Adding support for "web modules" may prove to be a more tractable problem.

Highlights of the MVP:

• All web modules are identified by URL.
• No file extension searching.
• No directory "index" support.
• Package.json export maps for mapping package imports to files.
• import.meta.require (or equivalent option) provides interop capabilities.
• --module/-m for loading the main file as a web module.

At the very least, it might be good to consider as a fallback in the event that various forms of transparent interop do not work out.

Looking forward to your thoughts and feedback.

Repository and prototype

[GeoffreyBooth]

Thanks for working on this! Do you mind summarizing some differences between the new ecmascript-modules implementation and yours? They both have some of those same bullet points.

[ljharb]

Using URLs seems like a big security and usability issue; such concerns have been raised before with deno’s focus in them. How would those work in an offline environment, since node can’t rely on anything but the filesystem?

[bmeck]

@ljharb the ESM loader in both upstream and fork is currently using URLs, you just whitelist the protocols allowed. Any unlocking of potentially hazardous URL types needs to be done with care.

[bmeck]

@zenparsing this is a pretty all encompassing proposal that doesn't discuss the reasoning for the bullet points or the relation to previous discussions and concerns for them. Could we get a more thorough explanation for some things like:

• All web modules are identified by URL. What do you mean by URL here. URL string format for cache key / resolution, any arbitrary URL?... etc.
• What happens to files without extensions / hashbangs since they cannot use flags
• "package.json export maps for mapping package imports to files." , all of this
• What happens for import() from other contexts (indirect eval Script / Function / CJS / etc.)?
• Why use import.meta.require vs the alternatives? +discussing previous concerns about it.
• How are formats determined/through what system can that be modified and/or virtualized? (asking because this seems to be a waterfall style attempt at a complete proposal of all features)

I think this issue would ideally be split up and made as separate issues against our minimal kernel because waterfall style aggregation of ideas doesn't seem to give us momentum or iterative feedback.

[zenparsing]

@bmeck Thanks for your (really good) questions. I'll need a day or so to follow up on each bullet point.

[Fishrock123]

similar to how node has been adding support for other traditionally browser-based features.

Yes, but, ideally when they make sense _for node_.
_(This has not always been followed much to my and some others' dismay.)_

Node.js isn't a web browser and, _in my opinion_, adding something that is purely due to limitations in a not-node environment should not need to be part of node.

That's my 2c, anyways.

[ljharb]

I actually don't mind adding a "web modules" feature; but imo that should be separate from, and shipped after, a node-first ESM system.

[weswigham]

A "web modules" compat filter is honestly just a lint rule applied on top of current node conveniences, IMO. Any number of tools or flags can do that lint, but there's no need for it to affect the default UX of node, IMO.

[jkrems]

A "web modules" compat filter is honestly just a lint rule applied on top of current node conveniences, IMO. Any number of tools or flags can do that lint, but there's no need for it to affect the default UX of node, IMO.

That is true as long as you never run 3rd party code. If you do, that changes. No amount of import maps will allow you to use that utility module that contains the line import findMatches from './find-matches';. Because that relative import will just plain fail if you're not running a special server or do bundling.

Tbh - if the end result is that node encourages publishing packages that won't work as-is in the browser even when they don't use node APIs, why not keep CommonJS? There doesn't seem to be a fundamental difference between "ESM that only works in node" and "CommonJS that only works in node". We can talk about "it's your choice!" all we want but if npm is full of modules with relative import statements that won't work in the browser, my choice doesn't matter.

[GeoffreyBooth]

There doesn't seem to be a fundamental difference between "ESM that only works in node" and "CommonJS that only works in node". We can talk about "it's your choice!" all we want but if npm is full of modules with relative import statements that won't work in the browser, my choice doesn't matter.

☝️ This. I thought the point of supporting ES modules in Node, of becoming spec compliant with regard to modules, was to make Node more similar and more interoperable with other environments. That's why "browser equivalence" is our no. 2 top priority just after spec compliance. Yes in some ways that makes Node worse, but people want to remove barriers between environments. There's some convenience lost if we ship without extension searching, but there's also convenience gained by most ESM npm packages being usable in browsers without transpilation. It's a trade-off. I understand many of the people in this group only code for server or Node environments, but the browser environment is very important to a lot of JavaScript developers, and what Node does affects that too.

[ljharb]

@jkrems due to the specification for ES Module specifiers, using import is a platform-specific API, because "how to resolve specifiers" is specified to be engine-determined. Separately, import maps can be produced - by npm or by node - that will allow your ESM node code to work in the browser without a build process.

@GeoffreyBooth one of the points is to implement the specification, of which some parts are implementation-defined. Browser equivalence is one of our priorities, but I think it's deceptive to call it "number 2" when we didn't agree on them in any particular order. Also, if removing a barrier makes node worse, then the barrier is a good thing.

[devsnek]

@GeoffreyBooth you can write all your code in your application to be browser compliant, but code in node_modules always and forever has to perform node specific resolution because of package.json and cjs and {list goes on}. So if you try to bundle node_modules without following all the rules that node follows, your server will not work. end of story. allowing ./file_without_ext doesn't make node less browser compatible because it's already impossible for it to be browser compatible. the only place you can affirm browser compatibility is your own application's code because you own the resolution of your own code.

[zenparsing]

Something else to ponder here: we've been thinking about this in terms of replacing CJS, but actually CJS is pretty great in different ways. I wonder: do we really need to replace all use cases for CJS?

To take an example: shebang scripts. I personally don't have a ton of experience with shebang scripts. I've mainly used them as very small entry points into a larger program (for instance, requiring in a module that implements a CLI program and lauching it). Do I need to write that shebang script with "web modules"? Can I just have a script that uses dynamic import?

#!/usr/bin/env node
import('./path-to-my-cli-program.js');

Just something to think about.

[GeoffreyBooth]

Do I _need_ to write that shebang script with “web modules”?

This came up when we were working out the edge cases for the import file specifier proposal. Specifically the case was I think running make lint on the Node repo itself, which runs tools/lint-js.js if I remember correctly. There’s no package.json in either the root of the repo or in the tools folder, so tools/lint-js.js runs outside of any package scope. An early implementation for IRP had such files running as ESM, which broke the Node build, so we changed these “loose” .js files to be defined as running as CommonJS (which I think was what the spec had said to do, but actually hitting the bug made us very aware of it). The part of the spec concerning symlinks, and making sure the module format to run the symlink path is based on the symlink target, came out of this case (and the real world example of /usr/local/bin/npm, which links to a npm-cli.js file). Your example of using import() could just as easily be achieved via a symlink, and the symlink has the advantage that it works in older versions of Node.

But yes, agreed, ESM doesn’t need to support _every_ use case that CommonJS supports; though since we added createRequireFromPath, many of CommonJS’ use cases (like importing JSON or .node files) can be handled via that. I’m sure there are still a few edge cases that can’t be achieved in ESM, and depending on how significant they are it might be fine to have some things where we just tell people “yeah, you need to keep using CommonJS for that.”

[jkrems]

Separately, import maps can be produced - by npm or by node - that will allow your ESM node code to work in the browser without a build process.

Let's please not keep repeating this, we have established further up that this isn't true. Unless I'm missing something important in the current proposal, it will not affect relative imports. So ./foo will not work unless the exact match exists, independent of any kind of import map. Any code in your dependency tree using that format will break in the browser, just like CommonJS files would. It is, in effect, just as (non-)portable as CommonJS.

[MylesBorins]

@jkrems afaict this proposal doesn't do extension searching... so relative modules in ESM should work identically in web platform + browser

since you cannot import CJS modules that would remain true for everything except for import.meta.require

Am I missing something

[jkrems]

@MylesBorins I was replying to @ljharb's comment which I read as in favor of (objecting to the removal of) extension searching.

Another nice benefit of the "No file extension searching" in this proposal is also the module identity issues. E.g. the following code:

import aJS from './a.js';
import a from './a';

In the browser this will always (?) lead to two different modules with their own execution and identity. If we do extension searching, I don't think there's a way we can keep this code consistent with the browser, even with a "smart" server or service worker. Unless I'm missing some important trick. Redirects will "fix" import.meta.url inside of the module but will still leave two separate modules for each requested URL.

[devsnek]

@jkrems so add a lint rule to your code for requiring extensions. I'm pretty sure such a rule already exists in eslint-import.

[jkrems]

so add a lint rule to your code for requiring extensions. I'm pretty sure such a rule already exists in eslint-import.

Right, I could lint all of node_modules but then that fails because the ecosystem is full of this code (given the current CommonJS usage patterns and that it would be allowed). Then... what do I do? Close the tab and go back to woodworking? ;) I honestly don't see how that solves anything. I don't think a project where all module code is written & controlled by myself is a realistic example.

[devsnek]

@jkrems you will never ever ever be able to put node_modules on a server and expect it to serve normally anyway, so your point of linting node_modules is kind of misleading.

[jkrems]

Why not, with the appropriate import map and for local development only? There's no reason why it shouldn't work if I use web-compatible modules in that code.

[devsnek]

@jkrems ...because cjs and .node exist?

[jkrems]

...because cjs and .node exist?

Right, but if the handful of packages my UI component uses do not use CJS or native modules (because... why would they?) that's not an argument at all. Saying "there's currently no way to author modules that would work in the browser, so it shouldn't be possible" for me raises the question why we are doing ESM in node in the first place. If the ecosystem this creates will only work in browsers if you use bundlers or compilers then... why are putting in a bunch of work to support ES modules? That seems to have the exact same properties as CommonJS at that point. Only that conditional imports are harder, so it may even be strictly worse..?

[devsnek]

@jkrems you can author modules that work in browser, but you cannot know that node_modules only contains modules like that, unless you scan node_modules with node specific resolution rules. (and even at that point you can't know for sure without better infrastructure like webpack which handles node builtins and such)

in the case of only installing things that are browser compatible, you still don't need to disallow people from using extension less because you can just not install those packages.

[jkrems]

in the case of only installing things that are browser compatible, you still don't need to disallow people from using extension less because you can just not install those packages.

Maybe we're talking about different things. What I'm saying is: Given the current practices, if we allow extension-less imports in node, nobody will bother to add extensions. If nobody adds extensions, all those packages will only work in node (or systems that emulate node, like bundlers). If that happens, what is the advantage of shipping ESM in node? I fail to see a single one.

[devsnek]

@jkrems I don't think the #1 reason for adding esm to node is to make node like browsers, it's because esm is a standardized module system that people enjoy using. personally I love esm because it won't run code unless the graph is valid, and that's just one feature of esm.

I suspect most cases of node usage don't have any reason to be constrained to browser rules. but that's ok; npm has rich ecosystems of both node and browser specific packages, and a large number of packages that work in both. (I believe that is the purpose of the isomorphic tag on npm)

[bmeck]

@devsnek

you can just not install those packages.

I believe this is what @jkrems and myself see as somewhat a point against shipping ESM at all.

@jkrems

If that happens, what is the advantage of shipping ESM in node?

Lets be quite direct. CJS has more functionality than ESM. So, why are we seeking to ship ESM? The answer is more complicated than "it is the future" and "it has pretty syntax". ESM is a a means to unify workflows, not just so that modules written for Node can work on the web, but also as a means to help ensure modules working on the web can work for Node. ESM will never have all of the dynamic richness that CJS has. Adding those features is problematic for any sort of consistent module system across these ecosystems. So, why was ESM added to the spec? To create a consistent and standardized module system.

When we talk about why we are implementing ESM, we are talking about moving to a standardized module system; ESM was not designed to have all the features of CJS, and adding all the features of CJS to ESM actively goes against the point of shipping ESM like @jkrems was attempting to point out. The more we diverge across ecosystems the less valuable ESM is. If we encourage divergence for the sake of existing divergence and status quo we actively go against one of the biggest points of ESM.

We should not focus on "this works in CJS, ergo, it must work in ESM", but rather on how ESM can benefit us by being a consistent ecosystem experience and a standard experience. To that end we are seeing other ecosystem environments like browsers attempt to support things that Node simply cannot remove like "bare" specifiers. We can allow people to opt into less standard behavior, in fact we should to allow them to have easier workflows! However, encouraging behavior that is non-standard across environments seems like removing one of the most appealing features of ESM.

ESM is not going to completely replace CJS, and as such it doesn't necessarily need to support all the cases that CJS has. Weighing if we can live without a feature in order to create a standard module system that works fairly well (maybe not entirely the same) across environments is the largest reason to ship ESM. If we are just concerned with syntax with can keep using tools to compile down to CJS. If we are just concerned with browser support, we can encourage using linters. I think the questions should not be about how to get features into ESM because CJS has them, but rather how to make ESM itself consistent. That consistency is the value!

People can already use the syntax today with compilation. Those compilers have all sorts of different configuration and options, which means that people using them must continue to configure them differently because they are not seeking to standardize across tools. We should not fall into the same pit of forcing everyone to deal with configuration differences, just because CJS has some behavior. ESM is not a clean slate certainly, but it is something we should be very careful about diverging for the sake of status quo.

[ljharb]

If it’s not going to largely/eventually be able to replace CJS, then to me that’s a point against shipping it. Two module systems is worse than one.

[MylesBorins]

@ljharb not shipping ESM is not an option for Node.js core. The ecosystem is adopting ESM and wants to see it happen. It has been standardized and adopted by browsers.

Not adopting ESM in core is the equivalent to calling it a day and saying "node is over" imho. It is not in my opinion an option and I would appreciate if individuals would stop using it as an ultimatum.

[ljharb]

@MylesBorins similarly, the reason to ship ESM in node is not solely “so we match browsers”, nor is it “so we exactly match browsers”, it’s “so we ship a standard that browsers are also shipping”, and the standard allows us to define our own implementation for specifier resolution. I’d also appreciate if individuals would stop using “browsers do it this way” as an ultimatum as well.

[MylesBorins]

I want to quickly apologize that my comment came across as directly calling out @ljharb. The intent of asking people to not use the "not shipping esm at all" ultimatum was aimed at the various people in this thread who made that statement. It's proximity to a direct response to ljharb was ill advised

[MylesBorins]

@ljharb regarding "browsers do it this way" is not my intent... rather encouraging platform independent and portable code. Browsers are the other major platform, but obviously not the only game in town. I do think it is extremely important for us to attempt to alleviate some of the need for build tooling in the ecosystem, that is part of the reason I would like to see features that are node specific be opt in rather than default.

[ljharb]

As youve mentioned before, you’re ok with install-time tooling - and i still think it’s entirely workable, even with the defaults i want, to generate drop-in import maps at install time, and require no further tooling. import maps work for all specifiers, not just relative paths, and not just bare imports.

[zenparsing]

Another thing to consider here is that (as mentioned briefly in the OP) designing a module system that completely replaces CJS in a seamless way and is compatible with web semantics and is forward compatible with new things we want to enable (e.g. a loader plugin that allows https specifier URLs!) may not be a tractable problem. It's certainly not feasible within the desired timeframes and it may not even be feasible at all.

Web modules, on the other hand, are likely feasible, viable, and desirable (although perhaps not exactly what devs are asking for).

[ljharb]

I agree with that, and i think we should first be shipping “node modules”, and follow up with “web modules” for the subset of node users that interact with browsers and want those semantics.

[weswigham]

The current import maps reference implementation would seem to allow mapping of relative imports, eg,

{
  imports: { "/node_modules/foo/bar": ["/node_modules/foo/bar.js"] }
}

Would allow a script at /node_modules/foo to import "./bar" and have it resolve to /node_modules/foo/bar.js. Given that the capability is there (and IMO, should be there), I'm not sure why extensionless imports (or index imports) would be a problem - their resolution can be serialized into an import map just like a bare import... imports even allows fallback URLs so you can even technically do extension fallback/searching by listing all the possible resolutions if for some reason you didn't know upfront which would be available (though you should know at the time the map is generated).

So again, I'm going to state that while browser modules are nice and all, and you can write modules in that style if you like, the tools aughta exist to be able to make all of your node style imports resolve in the browser anyway, so why make local development more cumbersome, and why force a change onto node users?

Like, if it ultimately comes down to "I believe extension resolution/index resolution in cjs was a mistake that I don't want to carry forward" for a lot of people here, I'd prefer it just be said that way. I and others would disagree, but at least it's a know point of opinion and not a technical restriction, which this is in the guise of.

[guybedford]

Let's not forget how much statting default extensions are costing us on Node.js startup time as well here. It's a performance and a browser argument.

In terms of generating the import maps, they will very quickly be polluted with lots of these extension rules if this is how the npm packages are populated. That's wasted bandwidth and statting time for what benefit? I'm still not sure I understand what the argument FOR automatic file extensions is apart from allowing dual mode without "exports" or other alternative entry point proposals.

[ljharb]

It allows transparent refactoring - with interop, from .js to .mjs and back - but also, from foo to foo/index to foo/blah (with a foo/package.json's "main" pointing to blah or blah/index) etc.

ESM is not an opportunistic moment for us to try to "fix" things that some of us don't like about node - if you want to improve node's performance, or package size (npm's problem, not ours), it should be improved for all parse goals, not just for ESM, and for all packages, not just for ESM packages.

[guybedford]

It allows transparent refactoring - with interop, from .js to .mjs and back

So is the argument that refactoring require('./dep') to import './dep' provides a significantly better user experience than import './dep.js'?

I personally don't see this as a big problem.

[zenparsing]

The ability to twist an import map around in order to override the relative specifier resolution logic of the browser is a bit beside the point, I think.

The solution has to be feasible. Very smart people have been trying to make "seamless" ESM work (with the constraints I gave above) in a way that makes everyone happy for years and it hasn't worked yet. I'm not that smart, but here's an example:

I tried to implement require-style resolution semantics on top of a no-transparent interop solution with automatic detection for the main script. It used "package.json/module" per the current convention and I thought it worked pretty well. It could even load things like rxjs as-is. And then I ran into a bunch of weird things at the edges:

• In order for the detection to work, the CJS loader had to either re-use "main" or be able to look at "module". Hmmm....
• What if a user loader changes the resolution behavior? Do we still load the auto-detected main file as-is or do we re-run the resolver? Hmmm...
• What if a user loader changes the translation behavior? Do we then need to run that main ESM file through the translation hook? What if that fails? Hmmm...

These are the kinds of problems that you run into when trying to blend the two module systems and support browser semantics like async loading.

For dynamic modules, we also have to factor in the problem of trying to get buy-in on loosening static module semantics at the language level and implementation support at the engine level.

Given our contraints, I encourage us to consider whether the "seamless" ESM experience is solvable within the timelines we have available to us.

[weswigham]

The ability to twist an import map around in order to override the relative specifier resolution logic of the browser is a bit beside the point, I think.

Considering the idea of browser compatible modules is only tasteful to _anyone_ with bare imports for modules, and those are _only_ possible with the import maps proposal, I do think the full capability of that proposal _is_ particularly salient. The proposal is capable of allowing you to define how _all_ of your imports will resolve - there doesn't seem to be a technical constraint here.

All I'm trying to get at is that this is a matter of _taste_, not technicality. And so long as that's true, my personal preference is for resolution in node's esm resolve to follow the cjs resolver as closely as feasible.

[zenparsing]

I definitely agree that we should not choose different resolution semantics based solely on taste.

I think we're all motivated by the goal of making things "just work" for users. But we might be in a situation where we're unable to give users a Babel-style module system and support other things we want like async loading, top-level await, automatic ".js"-as-esm, etc.

There's another prinicple I like which is: if you can't make it "just work", make it simple.

[GeoffreyBooth]

@weswigham Like, if it ultimately comes down to “I believe extension resolution/index resolution in cjs was a mistake that I don’t want to carry forward” for a lot of people here, I’d prefer it just be said that way.

I believe extension resolution/index resolution in cjs was a mistake that I don’t want to carry forward.

I mean, I don’t think there should be extension searching in ESM for more reasons than that, of course. But the performance reason alone is why this was a mistake: it scales poorly. The more file extensions Node supports, the worse the performance gets. It might’ve made some sense back in Node 0.3 in 2010 when require.extensions was added and no one imagined we’d ever someday have .mjs and .wasm and whatever JS AST lands under etc. But now that we know the direction that things have gone and are going, I don’t think the same decision would be made today (as it wasn’t in Deno).

Back in 2010 folks were doing things like transpiling CoffeeScript at runtime, and even in the CoffeeScript community this has been strongly discouraged for many years. I view extension searching the same way: it’s something that should be optimized out at build time. I acknowledge the convenience it offers, especially for transpiled languages, but Babel or some other transpiler can convert import './file' into import './file.js' at build time. I guarantee that if we ship ESM without extension searching, someone will write such a Babel transform within hours. And that’s fine! That’s the way it should be. And just as you _can_ transpile CoffeeScript at runtime, someone will write a loader to allow extension searching at runtime. That’s also as it should be. Users should need to opt in to a performance hit, especially since in this case there’s no way to opt out of it.

[ljharb]

That same argument applies to people who want to write ESM in files that don't end in .mjs - a build tool can do that for them; there's no need to slow down node, or add complexity to node, to support it.

[weswigham]

it scales poorly. The more file extensions Node supports, the worse the performance gets

And the community has developed tools for that, like baking the resolved graph into the resolver at build time using tools like yarn-pnp. If you wanted to pull a cache like that into node core, that'd be fine with me, too. (Although custom v8 startup snapshots are IMO probably the real best solution for many usecases from what I've read) That's a great example of solving the problem for everyone using all of node, rather than altering the core of how node works (or is percieved to work) for esm only to try to avoid it for a theoretical future fraction of the community. It's harming the whole ecosystem in pursuit of an uncertain goal, IMO. _Because_ node already does path searching and it's a community norm, I don't believe in usurping that norm for almost any reason, in any environment.

And if esm was differently spec'd, such that loading didn't have it's graph setup step that necessitated it's own resolver, we _would not be having this conversation_, because it would be _unconscionable_ that the addition of a new format (akin to .json or .node) could fundamentally alter basic norms of how resolution occurs. It is _only_ because technical restrictions force us to use a secondary resolver that we can even entertain this discussion - and I personally think that doing so is intellectually dishonest. esm may represent new syntax, and may technically require hoops to resolve to spec, but that doesn't fundamentally change how it should be presented compared to other code inputs to node, IMO.

[GeoffreyBooth]

And the community has developed tools for that, like baking the resolved graph into the resolver at build time using tools like yarn-pnp. If you wanted to pull a cache like that into node core, that’d be fine with me, too.

If Node is to have extension searching at all, that’s how it should be implemented—not just in ESM, but in CommonJS too. We shouldn’t be adding support for extension searching in ESM using the same problematic method that CommonJS currently uses.

My other reason for not wanting it in ESM is UX. I’ll leave debates over import maps and node_modules to others; what I care about is the code that users type, and therefore what they have to learn in order to type correct code. If Node supports extension searching, then users need to be aware that if they leave off extensions then their code won’t work in browsers (at least without build steps or a built import map etc.). Divergences are bad for UX, and the point of adopting the standard was to improve convergence between the environments.

To that end, Node should eventually support import statements accepting full URLs starting with https (and maybe http). Deno already does this, and it implements a cache like you’re describing @weswigham. Then code would be much more portable and interchangeable between Node and browsers; but it would also be even worse UX if extension searching worked for file:// URLs and relative URLs that resolved to files, but not for URLs starting with http and relative URLs resolving to network paths.

[devsnek]

@GeoffreyBooth

I believe extension resolution/index resolution in cjs was a mistake that I don’t want to carry forward.

good thing we respect other's use cases even if we don't like them. as an example, i have a strong dislike of most of the designs and constraints you've brought to this group, but i still respect them and deal with them, because you are a user of node and its my responsibility as a maintainer of node to respect your use cases at face value.

the performance argument is kind of irrelevant because a) since the new impl is c++ we can pretty easily parallelize it and b) i am subscribed to every issue opened against any repo in the nodejs org. i have seen maybe one or two issues about resolution speed, always in conjuncture with spinning a new node process up for each request or whatever, which is a gross misunderstanding of the node architecture. even if you want to misuse node, like wesley says, there are tools to fix this.

to that end, Node should eventually support import statements accepting full URLs starting with https (and maybe http)

based on past discussions i find it near impossible that the security wg approves this, so i wouldn't count on that as any sort of way we should be tuning UX.

[weswigham]

but it would also be even worse UX if extension searching worked for file:// URLs and relative URLs that resolved to files, but not for URLs starting with http and relative URLs resolving to network paths.

Just sayin':

{
  imports: { "jquery": ["https://cdn.js/jquery", '"https://cdn.js/jquery.js"', "https://cdn.js/jquery/index", "https://cdn.js/jquery/index.js"] }
}

is a tad ugly but certainly possible in the browser under the proposed map spec. the one thing that _isn't_ possible is anything using a package.json for resolution, which we are _absolutely_ tethered to and forced to do, so there is _no way_ you'll be able to make a multipackage project without something generating an import map, and since you're generating an import map, you may as well generate a full and complete one and support all of the niceties node is capable of offering.

[jkrems]

you may as well generate a full and complete one and support all of the niceties node is capable of offering.

That's cheap to say but we're talking about 100s of thousands of entries, depending on the size of the project. It would mean sending every single filename in node_modules to the browser multiple times over (at the very least twice, often a handful of times). So multiple megabyte of import map that need to be included on every page load. As opposed to 20k or something without full search resolution.

[MylesBorins]

Large import maps would also likely block render... so this isn't really a solution. Just because it "is possible" doesn't mean it would ever be used in practice

[GeoffreyBooth]

@weswigham I don’t understand your “Just sayin’” example. Sure, if I control the code that I’m importing from the Web, I could make sure that it has an import map that resolves extensions; but I’m pulling from some public source like unpkg.com? If I controlled and were publishing the code I’m importing, I’d probably be loading it from my local disk.

Both browsers and Deno support code like

import BoxGeometry from 'https://unpkg.com/[email protected]/src/geometries/BoxGeometry.js'

I think it would be short-sighted to think that Node wouldn’t support such code someday. There is already competitive pressure to do so.

Also, that example came from https://github.com/unpkg/unpkg.com/issues/34, where a user complains about how when they try to use an import statement to pull in an ES module from unpkg.com, it fails because the above file contains code like

import { Geometry } from '../core/Geometry';

and the browser can’t resolve it. The response from unpkg.com is that the user should change the URL to https://unpkg.com/[email protected]/src/geometries/BoxGeometry.js?module, where the ?module is a clue for unpkg to rewrite all the URLs in the import statements to have full extensions.

So yes, the ecosystem can work around Node’s divergence, but I don’t find this a good user experience; and it’s a burden on the rest of the JavaScript ecosystem.

[weswigham]

Large import maps would also likely block render... so this isn't really a solution. Just because it "is possible" doesn't mean it would ever be used in practice

If you care about bytes shipped, you'll never use import maps at all because all bytes in an import map are extraneous compared to instead rewriting all your imports (and paths-on-disk) to the fewest bytes (ie, 1-2 characters per) with tooling. Import maps are definitely a convenience feature for devs coming from nodejs.

[weswigham]

Also why would an import map block render on it's own? Wouldn't it be async and not block until a render-blocking script that uses an import (and so depends on it) exists?

[weswigham]

but I’m pulling from some public source like unpkg.com?

You'll need to pull an import map associated with it, too (or have precalculated one yourself). Otherwise how will you map it's transitive non-relative-path dependencies, anyway? You _need_ import maps at every step. And once you're using them, I don't see a reason not to use them fully.

[MylesBorins]

@weswigham import maps are not composable

[GeoffreyBooth]

Import maps can provide a workaround, yes; but so can configuring one's server to serve file.js for file. If I control the server and can configure it, or I can put an import map there, or just transpile the js for browsers, this isn't an issue. But users want to use code from URLs they don't control, like unpkg.com or jsdelivr or GitHub. If the author of the package hasn't made the extra effort to make their ESM code browser compatible, or included an import map that resolves extensions, then anyone trying to use it in a browser won't be able to. Browser compatible should be the default, not an extra build step.

[ljharb]

@mylesborins i believe they are composable in the latest spec.

[zenparsing]

Following up on some comments/questions upthread.

@Fishrock123

Node.js isn't a web browser and, in my opinion, adding something that is purely due to limitations in a not-node environment should not need to be part of node.

I hear you. I think one of the main reasons for node's success was that, going way back, the design effort was strictly focused on creating a really good server environment, instead of making compromises for the sake of compatibility with the web platform. The pendulum has swung the other way now, but I think that we should remember this design history.

On the other hand, the web has lead the way with (non-transpiler) ES module implementations and at this point I'm not sure that it will help to have effectively 3 module systems (CJS, ESM-web, ESM-node) rather than 2. (To clarify, I'm counting ESM transpiled to CJS as CJS.)

@bmeck

All web modules are identified by URL. What do you mean by URL here. URL string format for cache key / resolution, any arbitrary URL?... etc.

It's the same idea as the current --experimental-modules / ecmascript-modules semantics: URLs are used for resolution and as the module loader cache key.

What happens to files without extensions / hashbangs since they cannot use flags

For the MVP, the default loader refuses to load files that don't end in ".js" or ".mjs". This could be relaxed in the future. For instance, we could decide to have the default loader assume file targets are UTF8-encoded JS module code, similar to how the CJS loader assumes CJS scripts.

For the MVP, hashbang scripts need to be CJS.

"package.json export maps for mapping package imports to files." , all of this

Bare package specifier resolution works by first splitting the specifier into "package name" and "inner path" parts. It then crawls up node_module folders looking for a folder with the correct package name with a package.json file inside. The inner path is then matched against the "exports" key in that JSON file, according to the exports map proposal.

• Example package.json
• Possibly buggy prototype algorithm

What happens for import() from other contexts (indirect eval Script / Function / CJS / etc.)?

In the prototype linked from the repo, scripts are compiled with a HostDefinedOptions that contains a (possibly undefined) URL string. The default loader uses that as the base URL for resolving imports. If there is no URL string associated with the script, then the current working directory is used as a base.

Why use import.meta.require vs the alternatives? +discussing previous concerns about it.

We could get into the weeds on this one, but the main requirement is that this mechanism should not require any "plumbing" steps to get working. Users should not have to go to SO to remember the steps to enable CJS require.

How are formats determined/through what system can that be modified and/or virtualized? (asking because this seems to be a waterfall style attempt at a complete proposal of all features)

Obviously virtualization of loading is not a part of this MVP, but the architecture I have in mind goes like this: each V8 context is associated with an optional module loader via SetEmbedderData. The default context is associated with the default loader. New contexts do not have a loader, and there is no way to give them one in the MVP. Post-MVP, the idea is that new contexts can be created with an "import policy": basically, resolve and load hooks. If an import policy is specified, then the new context will be associated with a new loader that uses that policy.

For the default loader, there should be a way (post-MVP) to override the import policy. I don't have strong feelings on the API for this, other than an observation that most uses cases would seem to be served well by a plugin-style API.

[Fishrock123]

On the other hand, the web has lead the way with (non-transpiler) ES module implementations and at this point I'm not sure that it will help to have effectively 3 module systems (CJS, ESM-web, ESM-node) rather than 2. (To clarify, I'm counting ESM transpiled to CJS as CJS.)

If you consider "ESM-node" to "not be ESM" then you're going to have 3 regardless. :|

[zenparsing]

To clarify, when it comes to core semantics, I consider transpiled ESM to be an ESM-like front-end around the CJS module system.

[ljharb]

There’s always going to be more than one (two including CJS) module system - import.meta is allowed to differ by environment, as is specifier resolution.

The way to cohere that is to more concretely specify the language to allow for all of its environments, not to constrain some of them to the limits of others.

[evanplaice]

If the build process is relatively straightforward, supporting multiple formats isn't such a big issue. Most older libraries have been running multi-mode packages (ie IIFE/AMD/UMD/CJS) for years. Ideally, the long-term goal is to have ESM-first sources that can be built to support CJS and -- if absolutely necessary -- legacy JS (Ie ES5).

If you haven't heard, the Pika Package Manager just came out. It provides an interesting approach to solving this problem. It tree-shakes all but the top-level imports of a package and generates a single source file for use in the front-end.

Having used JSPM/System.js in the past, source maps work great for development but the overhead of requesting an entire tree of dependencies is too bad for prod; making prod bundles a necessity.

If we can get back to a mode where per-package (ie rather than split build bundle blobs) are the norm, then we'll we may see use of caching and loading libraries from centralized CDN sources (ie with local fallbacks) again.

As far as the Node side of that goes, as long as Node.js provides native ESM without any breaking quirks (ie .mjs required, alternative usage of pkg.module) then cross-platform compatibility shouldn't be an issue.

[GeoffreyBooth]

As far as the Node side of that goes, as long as Node.js provides native ESM without any breaking quirks (ie .mjs required, alternative usage of pkg.module) then cross-platform compatibility shouldn’t be an issue.

Can you please explain this part? What would be a breaking quirk that would imperil cross-platform compatibility, especially with regard to "module"?

[devsnek]

Your results will only include packages that are built with modern ESM syntax & include a defined "module" entry point in their package.json manifest.

damn shame cuz most of these are intended to be used with webpack/babel (non-spec esm) anyway :(

[GeoffreyBooth]

damn shame cuz most of these are intended to be used with webpack/babel (non-spec esm) anyway :(

Yes, this is what I was getting at with suggesting that Node ships defaults that encourage NPM packages to be browser-usable without rewriting. Obviously services like Pika and unpkg and jspm.io can rewrite JS on the fly to add missing file extensions or bundle files together and so on, but I don’t think it’s a good thing for users to be relying on third-party services for things like that. As you can see with Pika, if we leave this design space empty others will rush in to fill it, and we might not like what decisions they make.

[devsnek]

@GeoffreyBooth 🤷‍♂️ i'm fine with letting others fill this space, i think it makes sense to let people define their own subsets of node's larger feature set. i think this group has a bit of an over-representation in people concerned about browsers, not that being concerned about browsers is bad. in the end though, a lot of code for node can't run in browsers anyway, so letting people define their own experience makes the most sense to me.

[evanplaice]

I think this group has a bit of an over-representation in people concerned about browsers

Sounds like you assume it's a binary choice between browser-or-server. Many of us work in both. Many of us develop/maintain libraries that serve both ecosystems.

In addition to client/server development I'd also argue that CLI tooling is equally as prevalent. The JS tooling ecosystem is arguably the most mature of any language at this point.

For many of us, unifying JS across platforms will put an end to years of pain trying to support a broken ecosystem of of almost-but-not-quite-good-enough module formats. Besides, structuring a public API with ES imports is just, nice...

[devsnek]

@evanplaice i don't think it's a binary choice. i also don't think "the capabilities of a browser" is a good place to draw the line for node. node can go further.

Many of us develop/maintain libraries that serve both ecosystems.

this is my point, that it's already possible. you just have to maintain the hygiene of isomorphism instead of node doing it for you.

[jaydenseric]

Around 85% of the code that I write or publish is isomorphic/universal. For the remainder, I don't want to have to configure different linter configs, etc. for each environment.

Universal JS is the future. Differences between browser and server environments are hugely disruptive. For example, I've had nightmares trying to test in a Node.js environment libraries for interacting with standard fetch, FormData, File, FileList APIs. There are lots of gotchas with making browser code not crash in SSR that would not exist if Node.js valued commonality with browser APIs.

We are about to see a new generation of tools for shipping native, standard ESM with URL import specifiers for browsers and Deno. If Node.js doesn't have the capability, the community will be forced to employ a range of additional clunky tools.

Don't discount how hard it would be for new devs to learn the differences between browser and server environments if something as fundamental as imports worked differently:

• Node.js: "In a browser, you can import React from a URL like this... but if you want to SSR, because Node.js is different you need to setup Babel to transpile the …" etc. etc.
• Deno: "Import React like this". It's the same everywhere; easy.

Node.js will fragment the JS community and miss out on a lot of the packages and tools that will be published for Deno and browsers if it doesn't support standard import URLs and standard APIs like fetch.

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

[devsnek]

deno has a bundler built into it so I don't think that's a fantastic comparison for universal js.

aside from that, I've pushed for more common APIs between node and browsers, but it's a slow process and things don't get done unless someone volunteers their time.

none of this is relevent to the topic at hand though:

as long as node supports the subset of behaviour that deno/browsers/etc support, it's fine for extra stuff to exist on top. I don't see anyone complaining about deno filling the ecosystem with typescript, which browsers obviously can't run.

[evanplaice]

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

@jaydenseric Keep an eye out, this group exists to address the issues you mentioned. We just reached consensus today and -- if all goes well -- will be shipping it flagged in about 6 weeks.

[GeoffreyBooth]

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

@jaydenseric Keep an eye out, this group exists to address the issues you mentioned. We just reached consensus today and – if all goes well – will be shipping it flagged in about 6 weeks.

To be clear, we’re not dropping CommonJS. But we are aiming to support ESM in ways that are more compatible with user workflows and browser equivalence. The latest plan is here.

[MylesBorins]

Can this be closed?