Modules: Proposal: minimal esm implementation

[MylesBorins]

Hey All,

Like everyone here I've been thinking about this a whole bunch. For a while I've been trying to come up with a minimal implementation that we can iterate on. Something that could offer a fast path to deflagging.

Features

• esm works with flag --experimental-modules
• must use .mjs as entry point for node binary
• no transparent interoperability
• import.meta.require for cjs interop in esm
• dynamic import for esm interop in cjs
• package-name-map compliance for esm specifier resolution
  
  
  
  • must provide full path to module
  
  
  • no support for importing directories
  
  

Try it out today

Repo: https://github.com/MylesBorins/node/tree/esm-kernel

Download: https://nodejs.org/download/test/v11.0.0-test2018070976df5841a1/

$ NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/test   nvm install v11.0.0-test2018070976df5841a1

warning super naive implementation, lots of room for improvement... but it shows off the desired UX and all the tests pass locally on my machine.

Upstream PRs

• import.meta.require
  
  
  
  • https://github.com/nodejs/node/pull/21317
  
  
• package-name-map compliance
  
  
  
  • https://github.com/nodejs/node/pull/21729
  
  

TODO

• improve implementation
• get tests working on all platforms
• build consensus
• ensure loaders works
• ensure vm works

Note

I have not opened a PR upstream to remove transparent interoperability as I want to be respectful of the current conversations going on within the group. While import.meta.require and limiting extensions are not guaranteed to land, I do think that they are worth discussing independently on their own merits.

[WebReflection]

Quick review from user perspective.

Not only the binary

You wrote "_must use .mjs as entry point for node binary_" but I've noticed .mjs is mandatory as entry point regardless it's a binary file or not. Since there is a mechanism to start ESM, I'd love to have a flag that forces ESM as in node --experimental-modules --esm index.js.

Overall working ...

Overall everything seems to work as expected:

• you can import fs from 'fs';
• you can import rand from './module.js'; where module.js contains export default Math.random();
• you can const {require} = import.meta; and bring in CJS as you go
• you can require('./module.c.js') with module.exports = import('./module.js').then(module => module.default); in it and it works as expected, also same exact module

import fs from 'fs';
import rand from './module.js';

const {require} = import.meta;

const readFile = require('util').promisify(fs.readFile);

Promise.all([
  readFile('./package-map.json'),
  require('./module.c.js')
]).then(([data, crand]) => {
  console.log(rand === crand);
  console.log(data.toString());
});

... but !

I am not sure I am doing it right with the package-map.json file, but I've used this content:

{
  "path_prefix": "./node_modules",
  "packages": {
    "flatted": { "main": "esm/index.js" }
  }
}

and there's no way I can import flatted from the previous index.mjs file via import {parse} from 'flatted';

The ./node_modules/flatted/esm/index.js is a 100% valid ESM file already working on browsers and also NodeJS (but you need to use CJS otherwise it fails).

Last, but not least

• I was incapable to bootstrap ESM via node --experimental-modules -e 'import("./index.js").catch(console.error)'
• I was unable to use __filename or __dirname and the import.meta wouldn't provide any alternative

Wishes

• the .mjs at this point is needed just as entry point, meaning it's not really needed if not to signal how an env should be parsed. I wish we could use a flag for that instead of an extension.
• any alternative to __dirname or __filename would be great. I know it's relatively trivial to have both on user land but the following seems unnecessary boilerplate

import path from 'path';
const {
  pathname: __filename,
  __dirname = path.dirname(__filename)
} = new URL(import.meta.url);

console.log(__filename);
console.log(__dirname);

[bmeck]

Can you define "transparent" interop here so that it doesn't get confused.

I remain strongly opposed to import.meta.require in light of my discussions on the related PR.

I would like to better understand the reasoning behind all of these choices rather than just seeing something that works. It would be nice to see a write up on what things were chosen not to be supported and why the use cases they provide can be served with this minimal implementation.

• I have a particular concern about not being able to import CJS/C++ that I have brought up before and consider it necessary for any minimal implementation due to ordering concerns. Explanation of how those concerns are alleviated still in this minimal approach would be good.

• This seems to imply a Major breaking change of requiring the entrypoint to be ESM once unflagged? How does this affect "bin" scripts that currently execute as CJS.

• The package-name-map compliance seems to be at odds with current discussions about migration patterns in https://github.com/nodejs/modules/issues/139 . It would be good to know how migration is expected to occur here.

[demurgos]

The package-name-map compliance seems to be at odds with current discussions about migration patterns in #139 . It would be good to know how migration is expected to occur here.

To clarify, the issue with the package-name maps is that it requires the specifier to have an extension. Unless there's is another way to branch on the module type used by the consumer, this rules out the patterns relying on "mjs + js" (see table in #139). This currently leaves only "Default Export" and its clunky API as a completely safe pattern.

[jkrems]

I really like the direction of going for a more minimal solution.

I agree with @bmeck that there's some open questions. But I'd like to address them from a perspective of "what is the use case". So "Loading of polyfill or APM instrumentation that is implemented in CJS" instead of just "ordering concerns". Because that will most likely inform how we talk about it. When refactoring an application with "random" CJS files, the order is not as problematic in the general case.

[bmeck]

@jkrems we have files for configuration and singleton modules at work that need to evaluate prior to others. They configure things like servers and database connection pools. These are outside of polyfill or APM concerns.

[jkrems]

But I assume (hope) that they don't set random globals that other files then just assume are initialized..? If they do, I would treat them as "polyfills" (mutate global state to expose additional APIs).

[jkrems]

For additional color, the following will "just work":

import fs from 'fs';
import randomLib from 'esm-lib';

const db = import.meta.require('db');

db.accessTable(); // db init will still happen before this

[devsnek]

for even more color, the following will "just fail":

import.meta.require('x_polyfill'); // only available as cjs
import "something_expecting_x";   // only available as esm

I really don't like the term "just works" because it leaves out all the things that don't "just work"

[jkrems]

One possible answer for "how would polyfills work in that world" would be "it requires an intermediate module", e.g.:

import.meta.require('polyfill-a');

await import('./run-app');

But that definitely reduce the abilities of modules that need to ensure a polyfill (no static import of actual code, pretty awkward exports).

[bmeck]

I hate the term "just work"/"just fail" because it shuts down discussion of what/why things act a specific way under a guise of it being plainly apparent. We should try and explain why things serve or do not serve a use case.

I also want to bring up concerns that I have expressed elsewhere about introducing CJS permanently to all ESM modules under import.meta.require. As shown by @jkrems above, it can be used as a workaround and may require wrapping in other modules; however, it does introduce a large migration behavior of relying on CJS mechanisms that won't work on the web rather than allowing the migration paths described in other places. I am unclear on what actual value it provides vs just allowing imports of non-ESM module types. I see no value, and instead see major problems with exposing it. Topics like how to get people to purely use import which may have a different resolution algorithm and even cache algorithm from require become much harder if you don't have a single migration path towards one or the other. As it stands, without guaranteed ordering I would not suggest moving a large codebase to ESM because I don't think the benefits are worth. Wrapping modules and using different syntax/APIs depending on the module type being currently written/consumed is just not a beneficial path to me.

I also only see import.meta.require as a problematic API versus alternatives, including just exposing the ability to create a require function. If we put it into the migration strategy we are actively telling people to use APIs that are not web compatible. The ability to import non-ESM is not a compatibility concern and has well defined migration strategies to keep all new code using import, and moving towards ESM.

I am unswayed by the claims of web compatibility problems with importing non-ESM, especially in the face of browsers wanting to include non-ESM such as HTML modules. The arguments about compatibility have not made direct claims about how the ability to import CJS directly prevents support for some feature on the browser platform. All claims have been around codebases being unable to run in all platforms, however that claim is easily disproven if the codebase entirely is using ESM. Once browsers encourage non-supported formats such as HTML the same claim would be applied that their module system is no longer compatible with Node, but we can once again easily disprove this by creating applications that do not use HTML modules. What is the claim for compatibility problems in the face of the issues with not supporting importing CJS above? The claim that a codebase does not work on a platform is a codebase compatibility concern, please describe the platform compatibility concern.

[jkrems]

The ability to import non-ESM is not a compatibility concern and has well defined migration strategies to keep all new code using import, and moving towards ESM.

I'm not sure I follow. import 'some-thing-that-happens-to-be-cjs' is just as incompatible with the web as import.meta.require('the-same-string'). One difference is that the latter makes it immediately obvious and sounds the alarm bells ("this files will not work in a browser").

[bmeck]

@jkrems import 'some-thing-that-happens-to-be-cjs' can migrate to be ESM instead of CJS without causing consumers to break. That code itself is not incompatible but the underlying implementation of 'some-thing-that-happens-to-be-cjs' is. That is why I am saying it is a codebase incompatibility and absolutely not a platform incompatibility. The platform does not prevent you from writing code that is compatible in all environments, the code provided by the application is what is providing incompatible code. However, import.meta.require does not allow the same form of migration to occur. Per your claim that one is more obvious than the other, we can see errors from parsing and most likely also from linking when you import non-ESM, it has the same affect of showing errors in both those cases.

[ljharb]

I really don’t understand why we’re seeing implementation proposals before we’ve finished discussing transparent interop (all of its definitions) and defaults.

[zenparsing]

I agree with @ljharb here. It's good to see this work, and @WebReflection 's feedback is valuable, but I think the waters are still pretty muddy. I think we should come back to looking at implementation alternatives after we've done some more work on our framework for evaluating implementations.

[michael-ciniawsky]

IMHO createRequire 'vs.' import.meta.require should be discussed further

createRequire

import module from 'module'

const url = new URL(import.meta.url)
const require = createRequire(url)

const cjs = require('./main.js')

import.meta.require

import esm, { ns } from './module.js'

const { require } = import.meta

const cjs = require('./main.js')

createRequire includes a small extra step to get CJS working in within a module, which is a good thing as it makes CJS usage less convenient and explicilty indicates that CJS usage is mainly thought of as a migration step and usage should ideally be temporarily

I second @WebReflection that a flag node --module main.js is preferable over a file extension (.mjs) to determine the parse goal of the entrypoint, especially if 'transparent' interop is removed for now and besides packages (bare specifiers) the parse goal is known then (import/import() (Module (ESM))/require() ('Script' (CJS)))

+ 1 on enforcing extensions and stop importing directories
- 1 on __filename and __dirname for import.meta.require as those likely will never be supported by browsers (get use/support outside of node) and are trivial to code

How would the story for packages look like (without relying on file extensions) ?

{
  name: '@scope/pkg',
  version: '1.0.0',
  main: 'lib', // node ...
  module: 'src/index.js', // node --module ...
  scripts: {
    "build": "babel src -o lib" // [ '@babel/preset-env', { modules: true } ]
  }   
}

?

Besides enforcing extensions and stop directory importing, is there a anything from the package.json (npm/node) side of affiars that may improve package-name-map compliance further (at least in the bsic case) ? Things like a default entry name (index|main).js e.g

<script src="module.js" type="module" packages="path/to/node_modules">

module.js

import pkg from 'pkg'

// Resolving
script.packages + pkg.name + (index|main).js

— node_modules
|— pkg
| |— (main|index).js 

or the like...

[devsnek]

a flag node --module main.js is preferable over a file extension (.mjs) to determine the parse goal of the entrypoint

the author of the file should always have first say over what kind of file they authored. node can provide things to override defaults but the default behaviour must always defer to the author of the file.

[jkrems]

which is a good thing as it makes CJS usage less convenient

I'd be cautious with that line of thinking. Making anything less convenient should never be a goal. Because people are really good to find a path of least resistance. So if you make something less convenient, people will find a more convenient solution. And it's almost never what you actually wanted them to do.

[zenparsing]

@MylesBorins In general, I think this is a good direction. How does resolution work when importing a package specifier?

• Does it look for the "main" key in package.json and load that file as ESM?
• Does it still support index at the package root?

[zenparsing]

There appear to be some bugs around package specifiers; it seems to be possible to import from a directory if the directory is contained within a package path (e.g. import "pkg/folder").

Other than that, the general idea seems to be: you can only import from a directory if you are using a root-level package specifier, and in that case the normal package.json and index rules apply, with the exception that the target file is assumed to be ESM.

Using this approach, an author could dual publish by including both "index.js" and "index.mjs", or by having two similarly named files and using an extensionless entry in package.json:main.

I think I would simplify things: make it possible to import from any directory, but only use package.json and not index. It's not web-compatible of course, but that's OK. The simplicity of the rule offsets that downside.

The other issue here is that dual-publishing still forces authors to use .mjs, and it forces the user to "overload" main with an extensionless path.

The community has already converged on package.json:module. Is there any reason that node's package.json lookup rules could not simply use "module" instead of "main" when attempting to resolve an imported entry point?

As @WebReflection mentioned, I would also add an executable flag for specifying the entry point format.

Summary:

• Allow importing from all directories, but only use package.json
• Use package.json:module for specifying ESM entry points instead of "main"
• Add a --module format flag to the executable

With those changes, we can easily support dual-mode packages for interoperability and we allow users to keep using the ".js" extension if they choose.

[MylesBorins]

If anyone would like to push commits on top of this to fix bugs, extend / change implementation, or anything else please do. Follow up with a comment and if I have issues with what's pushed we can talk through it.

edit: oh yeah this isn't a PR... lol whooops. If you post a link to a commit sha I'll cherry pick and push first and ask questions later 😄

[guybedford]

I’m tempted to say the implementation approach is just about right (re package specifiers) - in
that having extensions for the main is ok, or allowing directory lookups
are ok, as long as it was resolved from a plain name to begin with thereby
being package map compatible.
On Thu, 28 Jun 2018 at 22:36, Myles Borins [e-mail] wrote:

If anyone would like to push commits on top of this to fix bugs, extend /
change implementation, or anything else please do. Follow up with a comment
and if I have issues with what's pushed we can talk through it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/141#issuecomment-401165179, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAkiyjuTrquyv7F3o91Wres8cq1HCLuHks5uBT5CgaJpZM4U62rR
.

[GeoffreyBooth]

In #137 there was a lot of discussion of initial entry points. I think the consensus was that we need a CLI flag to handle the --eval and STDIN cases, where there’s no filename to cue Node as to the parse goal. @bmeck made several good points about being careful to design the flag to avoid breaking WASM and other future module types.

[WebReflection]

FWIW, since it's actually super trivial to have filename and dirname in both node.js and browsers, I agree exposing those anyhow is unnecessary.

const {
  pathname: filename,
  dirname = filename.replace(/\/[^/]*$/, '')
} = new URL(import.meta.url);

that's already a cross env KISS approach.

[demurgos]

@WebReflection
Just to be sure, is it completely equivalent? What about some edge-cases such as Windows absolute paths or url-encoded parts in the filename? I remember seeing another expression to compute these from import.meta.url but it looked more complicated (I was not able to retrieve it).

[WebReflection]

What about some edge-cases such as Windows absolute paths or url-encoded parts in the filename?

AFAIK the import.meta.url is normalized in Windows too, since file:// urls are already accepted there, and since nobody ever bothered in node.js to require(window ? "..\\file" : "../file") I don't think there's any risk.

Of course, if you create folders or file names with encoded slashes maybe you are doing something wrong but yet, passed encoded, should be rightly understood via the URL.

However, this discussion is not about the best way to retrieve file and dir, so let's not bother others with that 'cause my point was to underline that it is, indeed, trivial, for the 99% of the use cases.

[guybedford]

Note the exact conversion from url to path requires (as detailed in
https://github.com/nodejs/modules/issues/121):

• decodeURI to be called
• the leading path separator to be removed in windows (/c:/ is how the
  pathname starts)

The above are common oversights everyone makes in assuming this which means
users will too.

As a result, supporting eg Chinese characters in module paths or supporting
windows will become common but reports for these workflows.
On Mon, 02 Jul 2018 at 13:05, Andrea Giammarchi [e-mail]
wrote:

What about some edge-cases such as Windows absolute paths or url-encoded
parts in the filename?

AFAIK the import.meta.url is normalized in Windows too, since file://
urls are already accepted there, and since nobody ever bothered in node.js
to require(window ? "..\file" : "../file") I don't think there's any
risk.

Of course, if you create folders or file names with encoded slashes maybe
you are doing something wrong but yet, passed encoded, should be rightly
understood via the URL.

However, this discussion is not about the best way to retrieve file and
dir, so let's not bother others with that 'cause my point was to underline
that it is, indeed, trivial, for the 99% of the use cases.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/141#issuecomment-401769326, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAkiyse6r0JUNQQDprOCqLq_xm7L5wFWks5uCf5vgaJpZM4U62rR
.

[zenparsing]

@MylesBorins

I've made the following tweaks here (comparison):

• Removed "no extension" error from lib and removed extension searching from module_wrap.
• Removed directory index searching from module_wrap.
• Modified package.json logic to use "module" key instead of "main" in module_wrap.
• Added --module and -m command line flag.
• Added node_modules support to import.meta.require.

Let me know what you think.

[zenparsing]

Out of curiosity I tried to create a project to see what would happen if I tried to import @angular/core with native modules (since that package.json contains a "module" field).

Using the build described above, it failed to import the dependency graph because a sub-dependency was not using file extensions in their import specifiers. I modified the branch to add back extension guessing, and was pleasantly surprised that I was able to import native ESM. 🎉

(Also, since angular uses named exports, it would have failed with automatic CJS-to-ESM conversion on their CJS main file.)

For transition smoothing, I think we should probably allow extension-less specifiers.

[WebReflection]

a sub-dependency was not using file extensions in their import specifiers

I think sticking with standards here would help thought, because maybe this was a lucky case where the file extension was missing, who knows which module points at a folder instead.

These libraries/frameworks heavily rely on bundlers, not on out of the blue native import from node, so giving them a direction that both node.js and bundlers can easily follow (i.e. explicit .js file extension) seems wiser in terms of migration through flags.

[ljharb]

"use an extension in the specifier" is not a standard - in the JS spec, in browsers, or in node (neither is "omit the extension in the specifier"; that's just a community convention)

[WebReflection]

"use an extension in the specifier" is not a standard

it kinda is, since we need a package map to address unresolvable names that won't exist by any other means.

Don't call it standard, call it must-have default fallback then, which is the standard/default behavior.

[ljharb]

There's nothing about a URL without an extension that makes it any more or less resolvable than a URL with one - browsers do not care about extensions, only the web server they make requests to, does (which sometimes will be node).

[WebReflection]

this makes no sense the moment the solution is to guess extensions which has nothing to do with URL resolution. I also find it funny you tell me extensions are not important to bring in ESM.

[devsnek]

@WebReflection when was the last time you want to a major website that has some form of ".html" on its landing page. (or any page for that matter, https://github.com/nodejs/modules/issues/141 doesn't have any extensions that i can see)

[ljharb]

@WebReflection i'm pointing out that there's no "standard" compelling forced, or disallowed, extensions - the only thing that should be weighed here is community usage.

[zenparsing]

I think the migration story is going to be more bumpy if packages that are already publishing ESM with "module" are required to update their specifiers to include the extension. I'm not sure that messaging will go over too well.

On the other hand, @WebReflection is correct that directory imports (found in ESM-published packaged) are still going to fail with this scheme. We would need some way to encourage those packages to modify their folder structure (perhaps by adding a package.json file with a "module" key in those folders).

I think whichever way we go, we would do well to develop tooling to help authors make their ESM compliant. I imagine that such a tool would:

• Flag usage of require, __dirname, and __filename
• Flag any directory imports that rely on index.* files

Fortunately, we have enough lead time that we can probably evangelize and nudge the ESM-publishing segment of the ecosystem toward compliant node ESM modules.

[WebReflection]

when was the last time you want to a major website that has some form of ".html" on its landing page.

I surely don't want my web server to start randomly crawling folders in search of something when the URL is www.fail.now/shenanigans, which is my point: currently shipped ESM behavior is quite strict in being represented by fully qualified names, including extensions.

[devsnek]

currently shipped ESM behavior is quite strict in being represented by fully qualified names, including extensions.

where is this behaviour? browsers are easily the largest shippers of this and in this context browsers don't have a concept of file extensions

[WebReflection]

I have no interest in this conversation because I am sure everyone understood what I meant: currently shipped ESM uses file system (JSC, SpiderMonkey) or static files (Servers) to load fully qualified names: no guessing around, unless of course, you cause the guessing by your own, but that's unnecessary (or another story).

[michael-ciniawsky]

@zenparsing It would be really appreciated if this could be made available behind --experimental-modules for the average Joe (like me :)) somehow to try it out and provide feedback.

Maybe leaving extension guessing etc in the first iteration? So ESM packages (like d3, angular etc) as of today may 'just work' ©️ (+ a warning that extensionless specfiers are under consideration to be removed in the future and discussion is here ?).

A warning is spammy for sure, but not sure how to kick off awareness/discussion about enforcing extensions with package authors otherwise 🤷‍♂️ ? There should definitely be a feedback loop of some sort for the community being able to voice their opinions about this.

Anways some sort of an implemention wise progress (with the possiblility to remove anything at any point in the future) would be a huge help to figure out remaining issues people might have (outside of the node/modules group), since folks can try it out and have the opportunity to provide a repro (in code) if anything is at odds for them.

Also, as an outside following individual, I'm personally under the impression that current discussion(s) about edgecase xyz are getting very noisy, time consuming and hard to follow without any concrete outcome ever surfacing from them (within the scope at the current point of discussion/implementation status) and sometimes, I'm sry to say it directly, I'm quite convinced they are even straight unproductive (e.g something with webserver(s) and how to config them is mandatory to make (simple) 'isomorphic' modules work at all (?) and MIME types (not saying the latter is not important, but I'm currently not following the need/gotchas there at all 🙃).

[ljharb]

I surely don't want my web server to start randomly crawling folders in search of something when the URL is www.fail.now/shenanigans

@WebReflection this is how webservers already behave - if shenigans is a file, they serve it, if it's a directory, they look for shenanigans/index.html (or dot-something). Additionally, many will ensure you have a trailing slash (or no trailing slash), or other redirects.

The "guessing around" in currently-shipped ESM implementations is on the server-side - just like it is for all URL-based web access whatsoever. Your server may choose not to do any guessing - but many do, and there is nothing whatsoever about ESM that precludes that.

[zenparsing]

@michael-ciniawsky

There should definitely be a feedback loop of some sort for the community being able to voice their opinions about this.

Absolutely! Perhaps @MylesBorins could merge my commits and provide an updated download link? 😺

[WebReflection]

Your server may choose not to do any guessing

I don't want guessing in my server. guessing means crawling, mean degraded performance, means complexity.

Which should I guess first, file.wasm or file.mjs or file.js or file.json or file/index.js, or maybe file/main.js and friends ?

This whole thread is about an MVP for ESM in modules and guessing is not part of an MVP, IMO.

[devsnek]

@WebReflection are you referring to the debug binaries for engines? I don't think we should be considering those at all... they just exist to test the engines.

[ljharb]

Migration is a required part of an MVP, and imo that includes guessing. In other words, we disagree on what "minimum" and "viable" mean.

As for which you would guess first, that'd be easy - you'd follow the order of require.extensions, as does (and will) every resolution tool in the ecosystem.

[WebReflection]

you'd follow the order of require.extensions

it's not even an Array and last one I have just checked is .node while I expect native extension (including .wasm) to have privileged meaning.

I also write ESM daily and I've never had any issue in specifying the bloody extension of a file name.

Bundlers solve that, good for them, but if you write ESM for real, you better write that extension or it wont natively work on browsers/server 'cause nobody writes overkilling rewrite rules for a missing .js, specially not CDNs so I don't understand what is your migration issue: bundlers users have no issues, pure ESM users write extensions already and also have no issues 🤷‍♂️

[devsnek]

package maps and bundling fix that. if you can somehow invalidate the existence of those then you have a point. no one expects have code written in node.js can be verbatim copied into a browser and just work beyond toy code and polyfills. idiomatic node lets you drop the file extension because what a package is written in is an implementation detail. we shouldn't give that up for misinformed browser compatibility.

[weswigham]

Which should I guess first, file.wasm or file.mjs or file.js or file.json or file/index.js, or maybe file/main.js and friends ?
This whole thread is about an MVP for ESM in modules and guessing is not part of an MVP, IMO.

I'm pretty sure the DirectoryIndex command in a .htaccess file configures that in apache. Not sure what the equivalent is in IIS. Express's serve-static has an option for it.

Implied extensions and a priority list therein are really common among webservers. You may have never reconfigured it yourself, but it's usually there.

Bundlers solve that, good for them, but if you write ESM for real, you better write that extension or it wont natively work on browsers/server 'cause nobody writes overkilling rewrite rules for a missing .js, specially not CDNs

unpkg is a fairly popular CDN which _does_ have that style rewrite rule. So they do exist.

[ljharb]

Also cdnjs

[WebReflection]

package maps and ...

if you use those, you won't write fully qualified path up to the extension. If you have package maps you import "module", you don't import "./some/path/to/file" omitting only the extension, so I keep saying there's no need to guess ever here, and it feels like many in here use bundlers and are solving problems that don't exist with bundlers.

Anyway, if you think omitting the extension is not just useless overhead and unnecessary for this MVP go ahead, all I was saying as daily ESM user is that explicit file names are not an issue, and never will be.

In JSC and SpiderMonkey these are also mandatory and ESM shipped like a year ago ... that is what I call a MVP: it works, it doesn't have much philosophy or guessing around file paths, it shipped ahead of time.

[michael-ciniawsky]

if you use those, you won't write fully qualified path up to the extension.

You surely do write it, in package-name-map.json :)

package-name-map.json

{
  "path_prefix": "/node_modules",
  "packages": {
    "moment": { 
       "main": "src/moment.js" 
     },
    "lodash": { 
         "path": "lodash-es", 
         "main": "lodash.js" 
     },
     "wrong": {
         "main": "src" // <= No
     }
  }
}

https://github.com/domenic/package-name-maps#basic-url-mapping

Note how unlike some Node.js usages, we include the ending .js here. File extensions are required in browsers; unlike in Node, we do not have the luxury of trying multiple file extensions until we find a good match. Fortunately, including file extensions also works in Node.js; that is, if everyone uses file extensions for submodules, their code will work in both environments.

👌

[WebReflection]

@michael-ciniawsky it seems like we agree, or at least, I agree with everything written in that quote, and also JSC and SpiderMonkey already work like that (fully qualified names).

[devsnek]

JSC and SpiderMonkey already work like that

you keep saying this, but what does it mean. are you referring to the cli debuggers they maintain? the APIs built into the engines? I have no idea that that means.

[WebReflection]

echo 'print(123)' > module.js
echo 'import "./module.js";' > entry.js
jsc -m entry.js
# prints 123

you can try omitting part of the name (extensions are irrelevant, imports resolve the path though)

echo 'import "./module";' > entry.js

and see that jsc -m entry.js now produces an error

Exception: Error: Could not open file './module'.
fetch@[native code]
requestFetch@[native code]
requestInstantiate@[native code]
requestSatisfy@[native code]
[native code]
promiseReactionJob@[native code]

Both JSC and SpiderMonkey, and browsers, uses out-of-the-box resolvable names, without any guessing involved, neither folder or extension crawling.

This is the meaning of _{^{(KISS and)}} an MVP, imo.

edit

and browsers

meaning, they ask for what you are importing, they don't crawl the server, they don't guess extensions, they ask for a module with that name, end of the story.

[devsnek]

so you are referring to the cli debuggers. those only exist to test functionality of the engines. i would request that you do not base any behaviour in node.js on those, as node isn't a debug utility - its a production runtime with millions of users. i'm also slightly upset about that as i'm currently working on some examples of module loading for V8/D8 and i would dislike if my work was misrepresented like this.

[WebReflection]

@devsnek you are ignoring SpiderMonkey, powering GJS and others, where it's the exact same. I am telling you how engines that shipped ESM already work, and also how the browser work.

If you don't want to listen, it's not me misrepresenting anything, it's you not wanting to listen, sorry.

I think we have done with this discussion.

[devsnek]

@WebReflection how GJS resolves stuff isn't specific to spidermonkey, but it is a better example of your point. i wish you had brought it up earlier :)

[zenparsing]

I think we've probably reached the end of this line. I think we can all agree that file-extension searching is a web/node interop issue. On the other hand, we need to balance that against the need to provide a smooth transition path to native ESM modules on node. We can debate how to balance those concerns later when we have more feedback.

[WebReflection]

how GJS resolves stuff isn't specific to spidermonkey

GJS uses js52 here and js52 behaves exactly the same. Swap jsc with js52 in previous example, omit the extension, then:

Error: can't open ./module: No such file or directory
Stack:
  fetch@shell/ModuleLoader.js:12:16
  loadAndParse@shell/ModuleLoader.js:18:22
  @shell/ModuleLoader.js:29:47
  Reflect.Loader<@shell/ModuleLoader.js:25:9

TL;DR I've used jsc 'cause I was on mac, not because it was anyhow more or less relevant to my point.

Also browsers do the exact same, they don't guess, they don't crawl, they ask one thing only (that might be changed by the map but that's another story, I have nothing against the usage of the map)

the need to provide a smooth transition path to native ESM modules on node.

AFAIK there is not much transition to smooth out since std esm already does that. This is what I mean when I say most developers here use bundlers or esm resolver and there's nothing to smooth out for them 'cause both bundlers and esm can guide developers to migrate start deprecating or warning in console without breaking.

Tools should be there to help migrating, following what Node.js ships.

This shouldn't be vice-versa, otherwise Node.js will be always doomed by tooling adoptions.

Sure thing though, we need feedbacks, but I am worried people that use tooling will create problems that don't really exists (and they'll keep using tooling anyway).

[devsnek]

@WebReflection they don't use the cli debugger though... they're using their own implementation of resolution and evaluation which you can read through here: https://gitlab.gnome.org/GNOME/gjs/blob/master/gjs/module.cpp

edit: i'm not even sure they're using ESM... if anyone is familiar with mozjs's api please let me know

[WebReflection]

i'm not even sure they're using ESM

it was brought up already months ago in the mailing list, which is why I keep saying we should keep it simple. GJS has live bindings but a completely different module system which, AFAIK, is going to be integrated with ESM.

Right now though, the import is limited and it will always add a .js extension, which is why it'd be lovely to keep it simple and have ESM with explicit extensions and let loaders and package maps handle all other cases.

MVP of ESM: explicit extensions, never ambiguous, faster to ship, less to discuss. You have loaders to solve everything else. How cool is that? 🎉

---

actually: loaders can help smoothing out migration !!!

[devsnek]

MVP of ESM: explicit extensions, never ambiguous, faster to ship, less to discuss. You have loaders to solve everything else. How cool is that? 🎉

super cool if we didn't have 750,000 cjs modules that are still first-class citizens in the node ecosystem.

if everyone has to use a loader (and they will have to use a loader) then why not make it the default...

[WebReflection]

super cool if we didn't have 750,000 cjs modules that are still first-class citizens in the node ecosystem.

those can be brought in via import.meta.require which works like a charm in here already. Have you tried this MVP? I suggest you do.

However, loaders can solve that too 🎉

if everyone has to use a loader (and they will have to use a loader) then why not make it the default...

Nobody has to use a loader here. I've tested this and I didn't need any loader. Also my code wouldn't need any loader. Package maps, require when needed, pure ESM straight forward for everything else.

Who's using ESM today is already behind bundlers or loaders, so they don't have to change anything 🎉

MVP

A minimum viable product (MVP) is a product with just enough features to satisfy early customers, and to provide feedback for future product development.

I see it this way:

• is the extension guessing mandatory to ship this? No.
• can everything related to the magic guessing be implemented regardless from users? Yes.

That's it. But I also stop now, since like I've said there's not much else to add from me.

Regards

[MylesBorins]

I've updated my MVP branch based on some comments in this thread (updating original post as well).

Changes include

• fix to import.meta.require for node_modules as found by @zenparsing
• no longer enforcing extensions. Changes have been made to the resolution algorithm to no longer resolve extensions or directories for local modules.

Repo: https://github.com/MylesBorins/node/tree/esm-kernel

Download: https://nodejs.org/download/test/v11.0.0-test2018070976df5841a1/

$ NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/test   nvm install v11.0.0-test2018070976df5841a1

[WebReflection]

@MylesBorins it looks like you dropped the need for the extension but you haven't put in the option to enforce ESM so that now we have ambiguity.

./module not found by import in ./index.mjs. Legacy behavior in require() would have found it at ./module.js

If I start from index.mjs and import from ./module.js, which is a perfectly valid ESM file, everything is OK. If I now import from ./module which is a perfectly valid ESM file, node assumes the .js version is not good, but it works without issues when explicitly written.

It is also impossible to bootstrap ESM via node --experimental-modules -e 'import("index.mjs").catch(console.error)' because the --module or --esm flag is missing, meaning an import always fails in --eval at this point.

Considering I don't care about extensions, since I've described why it's a footgun to omit those but I like the moment I don't everything works as expected, would you be so kind to bring in the --module or --esm or --type=esm or --type=module flag to this MVP ?

Thanks.

[GeoffreyBooth]

Besides the above reasons, we need a CLI flag to enable ESM mode so that code like this works:

node --module --eval 'import path from "path"; console.log(path.sep);'

[WebReflection]

FWIW, other CLIs use --module shortcut'd as -m to force-enable ESM over JS.

Since other extensions don't really need any enforcement and have no differences between ESM env or not (i.e. .json or .wasm or even .css), I wonder what's stopping node to add such flag.

Behavior

Everything evaluated as JS, or with a JS oriented extension (both .js and .mjs), is loaded as ESM unless import.meta.require is used.

[MylesBorins]

Closing as this is quite similar to what we've ended up doing for the minimal kernel