modules 🚀 - disambiguate "web compatibility"

I would call import 'cjs' a code compatibility concern, just like __filename or accessing node-specific globals like Buffer. In all cases it's valid syntax but will do different things in the browser (in most cases: it will break in the browser). You can change another part of your project to "fix" it but at that point you changed the code and it is no longer import 'cjs'.

Magically switching import 'cjs' once a library exposes a module also has the disadvantage that we get one of two undesirable outcomes:

Libraries need to keep exporting default as an object containing a copy of their named exports.
Libraries need to treat adding module support as a breaking change.

There's an implicit third outcome, equally undesirable, where adding support for <insert mechanism that allows named exports for CJS> is a breaking change or forces a weird reserved default export.

jkrems on 28 Jun 2018

😕1

@jkrems can I ask for examples that mandates people to import CJS, or presents a path that prevents them from shipping ESM?

What you describe is that there a lack of feature parity between the platforms, I claim that is not a compatibility concern since nothing enforces people to do something that must be incompatible with the web.

You are describing a migration problem as well, but you are not stating anything that prohibits people from shipping a full ESM graph to the web. Do you have examples of how allowing import of formats not supported by the web prevents importing web supported formats?

bmeck on 28 Jun 2018

@jkrems maybe to clarify a bit, what about import 'cjs'; has different aspects than import.meta.require('cjs') that makes import.meta.require able to avoid breaking module graphs shipped to both platforms.

bmeck on 28 Jun 2018

what about import 'cjs'; has different aspects than import.meta.require('cjs') that makes import.meta.require able to avoid breaking module graphs shipped to both platforms.

Nothing. It just makes it immediately apparent that a file containing import.meta.require('some-lib') won't work in the browser. For import 'some-lib' I have to start researching and/or reading some-lib's code.

jkrems on 28 Jun 2018

👍1

@jkrems would the notification of linkage failing and certain parse failures not serve the same purpose? In addition as I mention above some-lib could take multiple routes to support shipping both formats. Is your concern just about wanting to define the format of the module being imported at the location it is imported?

bmeck on 28 Jun 2018

👍1

I’ve thought it was odd that Node allows import './file' but browsers require the full filename including extension, e.g. import './file.js'/.mjs. Ditto for folders resolving to /index.mjs. Regardless of whether you think this is behavior that Node _should_ allow in ESM, I think this automatic resolving of file extensions and of folder entry points is something that should be on the list of code compatibility concerns.

I think there’s a compelling argument to be made for dropping automatic extension resolution and automatic folder index.mjs discovery, in the interest of browser compatibility. The use case for those features, it seems to me, is import interoperability where Node can automatically resolve to .mjs or .js, or index.mjs vs index.js. That’s a valuable use case, to be sure, but I wonder if it shouldn’t be something that users opt into by adding a loader.

Such an approach would pair with the package name maps proposal to provide ways to do things like import 'lodash' in both browsers and Node. Assuming that gets adopted, lots of NPM modules might be otherwise perfectly web-compatible aside from their dropping of file extensions in import statements, for example. It might be a good idea for Node to nudge them in the direction of browser compatibility, by somehow making them opt in to the browser-incompatible syntax.

GeoffreyBooth on 28 Jun 2018

@GeoffreyBooth I've put the resolution algorithm into Platform compatibility concerns.

bmeck on 28 Jun 2018

I'd also note that even with the example of import 'lodash' which doesn't have an extension it would still be able to resolve to any format, so it is separate from the ability to import non-ESM.

bmeck on 28 Jun 2018

@jkrems you have to do that research regardless, because any module anywhere in the graph might use fs. Are you suggesting that ESM shouldn't be able to import node's core modules?

ljharb on 29 Jun 2018

@GeoffreyBooth a) browsers don't mandate the extension, they use URLs. if the URL lacks an extension, so too can the import path. b) with package name maps, you'll be able to omit extensions in browsers (just like is the best practice in node/npm), and it will work the same without a build step. This is a good thing.

ljharb on 29 Jun 2018

a) browsers don’t mandate the extension, they use URLs. if the URL lacks an extension, so too can the import path.

I don’t quite follow. I didn’t mean to imply that browsers care about extensions, I only meant that they require fully resolvable paths/filenames (yes, as part of a URL). My example import './file.js' is using a relative URL, like <script src="./file.js">. If the URL lacks an extension, doesn’t that just mean that the webserver needs to decide what to serve for that? So for import './file', the webserver would need to either serve an extensionless file named file or know to serve file.js, the same way that many webservers automatically serve index.html for folders?

Since it’s not standard for webservers to resolve an URL ending in file to file.js or file.mjs, it feels like something that Node shouldn’t do either, at least not by default. If it _becomes_ standard, such as via package name maps, sure, that would be great. In that case, though, I would think that Node should do it the same way, through package name maps, rather than its own custom implementation.

GeoffreyBooth on 29 Jun 2018

It's something node already does by default, and it's something users expect. Whether it's implemented in terms of package name maps or not, I think that it would be extremely hostile to users to omit.

ljharb on 29 Jun 2018

I'm going to modify the pr I opened for file extensions.

Seems based on this conversation that the appropriate behavior would be to
load the path without any resolution, instead of enforcing file extensions

On Thu, Jun 28, 2018, 6:30 PM Jordan Harband notifications@github.com
wrote:

It's something node already does by default, and it's something users
expect. Whether it's implemented in terms of package name maps or not, I
think that it would be extremely hostile to users to omit.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/142#issuecomment-401192567, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecV0KevcwNdZ3LQHGyEYapGdOGNSuAks5uBVjugaJpZM4U7_X5
.

MylesBorins on 29 Jun 2018

😕1

@ljharb I hear what you’re saying, and the user hostility is why there should still be some way for them to do it: via a loader, via package name maps, or something else. But put simply, either we’re trying to be equivalent with browsers by default or we’re not.

The current behavior certainly wouldn’t change in CommonJS, it’s only ESM we’re discussing here.

I’ve come to accept that Node isn’t going to support everything that current transpilers do, at least not without loaders or other hacks/patches. I think the explanation that “Node has added support for import and export in the same way that browsers support that syntax and ES modules” is a compelling explanation that most users will grasp, and helps defuse complaints about things that don’t behave as users expect or might want.

GeoffreyBooth on 29 Jun 2018

I don't think we should be trying to be equivalent by default with browsers - browsers don't have filesystem access, nor a massive CJS ecosystem it needs to retain compatibility with _{(they have a much more massive legacy ecosystem to retain compatibility with)}.

ljharb on 29 Jun 2018

❤1 👍1

But put simply, either we’re trying to be equivalent with browsers by default or we’re not.

I'd note that in all scenarios we are not equivalent. using import.meta.require() is the same level of concern as supporting import 'cjs'; since it also won't work in browsers because the dependency is an unsupported format, not to mention that import.meta.require won't even be shipping in browsers. There is no scenario where we have this false sense of equivalence.

bmeck on 29 Jun 2018

Is there any advantage to choosing import.meta.require versus import "cjs"?

import.meta.require is less complicated to explain
import "cjs" can give the user some surprise, if they are expecting named exports
import.meta.require nudges authors to intentionally opt-in to an ESM API, and therefore may encourage migration

zenparsing on 29 Jun 2018

😕1

I think the distinction is that, we don’t expect equivalence with browsers whenever CommonJS or interoperability with CommonJS is involved; but in all-ESM mode, I would expect Node to behave the way browsers do. If I can do import './file.js' in a browser, I should be able to do the same in Node. If I can’t do import './file' in a browser, where the file to be loaded has an extension, I wouldn’t expect Node to let me do so (without patching).

GeoffreyBooth on 29 Jun 2018

i think thats a bit of a red herring. both node and browsers have non-esm formats they would like to interact with (wasm, html, cjs, c++, etc).

devsnek on 29 Jun 2018

👍1

@zenparsing imo requiring intentional opt-in actually discourages migration :-/

@GeoffreyBooth you can't use document in node; i don't think it's reasonable to assume that "i can do it in a browser" means "i can do it in node", and certainly not the reverse.

ljharb on 29 Jun 2018

👍4

i don’t think it’s reasonable to assume that “i can do it in a browser” means “i can do it in node”, and certainly not the reverse.

Sure, not for everything. I don’t expect Node to support document, as there’s no document in Node. But there’s no reason Node can’t support import './file.js'. The goal is browser equivalence. Just because there are zillions of examples you can point to of places where Node and browsers diverge doesn’t mean that we should add more if we don’t have to.

Let’s cut to the chase: @ljharb, I assume you want to make import './file' work because you want file-level import interoperability where import can import both a script-goal .js file or an ESM .mjs file. Right? And/or the convenience of leaving out extensions, as is common in Node. Any other reasons?

And so the question is whether those goals are more or less important than enforcing browser equivalence when it comes to the import statements that users type in JavaScript code that could potentially execute in either Node or browsers. We have conflicting goals here, and so it’s a subjective call of which priorities are more valued.

GeoffreyBooth on 29 Jun 2018

That's the primary reason, yes - and I think it's much more important. Package name maps allow the browser to work as ergonomically as node does here, which is great.

I'm also happy to have node support package name maps in some form; but I think it's critical that the default implementation work like node already does.

ljharb on 29 Jun 2018

❤1

@zenparsing those have counterpoints so it doesn't really aid:

import.meta.require is less complicated to explain

requires knowing the difference in the algorithms, supported formats, loaders, and caches. i presume this is more burden than learning how import treats specific formats.

import "cjs" can give the user some surprise, if they are expecting named exports

clear linking errors removes that point, same as when you import anything that doesn't have a specific named export. surprise is easily fixed.

import.meta.require nudges authors to intentionally opt-in to an ESM API, and therefore may encourage migration

import.meta.require guarantees your source text cannot be used on the web platform. there is no migration path with it.

The only point that seems to have a bigger impact that its counterpoint is the one about surprise and named exports. However, it has clear errors and easy to learn either by documentation or experimentation using import * as or import().

@GeoffreyBooth you make a point about module resolution but nothing about loading CJS. If you resolved to ./file.js and it was served with application/node it would error, but nothing prevents you from loading ESM on the web by letting Node load application/node. The same error of being unable to load a dependency due to it not being supported by browsers applies if you use import.meta.require or if you have any file that imports a node core library like http.

In addition, you can load ./file:

// this line won't work on the web
// linkage fails so this file doesn't even evaluate
// under the argument for being unable to import things the web cannot it is a compatibility problem
// we should use `import.meta.require('http')`
import http from 'http';

// everything in here doesn't work because the web also doesn't ship these APIs
// we once again are not equivalent
http.createServer((req, res) => {
  let files = {
     '/': {type: 'text/html', body: `<script type=module src=/file></script>`},

    // no extension required, just MIME
    // can also do similar using .htaccess / CDNs / cloud storage providers like S3 / ...
    // <FilesMatch "^[^.]+$"> // change as needed
    //   ForceType text/javascript
    // </FilesMatch>
     '/file': {type: 'text/javascript', body: `console.log(1)`},
  };
  let file = files[req.url];
  res.writeHead(200,  {'content-type': file.type});
  res.end(file.body);
}).listen(9090);

addendum: fun fact! even if you set *.js to application/node it will load as Script on the web when using non-module type <script> tags! <script type=text/javascript> doesn't check the content type it loads, at all!

bmeck on 29 Jun 2018

👍1

@bmeck I didn’t mean in my example that file.js was to be assumed to be CommonJS or Script, forgive me for leaving that out. I said above that I was describing an all-ESM environment, but I should’ve made extra clear that I was intending file.js to be an ES module. I understand that’s confusing in this context since in experimental-modules ES modules must be files ending in .mjs. What I had in mind were the examples of how to use import in browsers, such as at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import, where you see lines like:

import * as myModule from '/modules/my-module.js';

Yes, obviously you can write lots and lots of Node code that doesn’t work in browsers, and vice versa. That doesn’t mean there isn’t value in trying to align with browsers where we can. This is one place we can, admittedly at the cost of preventing other use cases that people really want (without loaders/patches). Maybe we’ll decide that browser equivalence here isn’t worth the cost, and we would rather make an exception to the browser equivalence goal for these other use cases that are deemed more important.

GeoffreyBooth on 29 Jun 2018

👍1

There seem to be a couple of fairly strong arguments against both import.meta.require and import "cjs":

import.meta.require injects legacy APIs into the new module system
import "cjs" (assuming no named exports) will probably frustrate users when moving from transpiled ESM to native ESM that are expecting named imports

The max-min implementation must provide some way to load CJS from ESM, but given that both are controversial we might need a third max-min option. Wasn't there a "createRequire" proposal or something floating around?

zenparsing on 29 Jun 2018

Also, a meta-note: I'm not really digging the GH emoji responses in the context of this debate. I find them emotionally distracting; I think they make it harder to come to a shared understanding.

zenparsing on 29 Jun 2018

Also, a meta-note: I’m not really digging the GH emoji responses in the context of this debate. I find them emotionally distracting; I think they make it harder to come to a shared understanding.

Strong +1 to this (almost used an emoji for it). People have a right to express themselves, of course, but 👍s to support one side of a debate only make arguments more heated. It feels like a lot of 👍s are given to express solidarity with one side or another, which doesn’t help forge consensus.

GeoffreyBooth on 29 Jun 2018

i like at least 👍 because it lets us gauge agreement without cluttering comments

devsnek on 29 Jun 2018

I think the frustration of "i can't have named imports from CJS, but oh well, i can still import it and destructure in another line" is way way less of a downside than legacy APIs in the new module system.

ljharb on 29 Jun 2018

I can try and expand to comments instead of emojis to give greater clarity in what I agree with within other comments, but already feel I occupy a large amount of text in multiple threads.

@GeoffreyBooth you do have a point with some documentation, but there are plenty of others with named exports from CJS, loading CJS using .js, and loading files while using Node's path resolution algorithm without extensions and using node_modules; some are even saying to use .mjs even if you are purely on the web platform now since Modules don't act like Scripts. I'm not sure having one specific set of documentation say one thing means anything of note when others say different things.

@zenparsing I'm not sure minmax makes sense here because the very nature of being able to load CJS in any form is the heart of the problem if you claim that Node should be equivalent to web browsers. There is not a solution with no downsides because of this. Minmax is about finding the solution with the minimal set of non-controversial features but the very nature of being able to use any of the CJS on npm or from Node core is the root of the claimed compatibility issues. The only solution would to be unable to load CJS or Node core because those are being claimed as incompatible with the web. I agree that they are not features that the web provides, but they have no problems preventing code to be written for both if we don't expose APIs that cannot ship to the web (notably any form of require()). Allowing people to import dependencies that are CJS is vastly different from forcing them to use web incompatible APIs, they have multiple migration paths if they can change the implementation of a dependency to ESM and can use things like service workers, package name maps, etc. to run on the web without rewriting their source text. Using loader provided synchronous APIs would not work with approaches like package name maps or even service workers.

Per createRequireFunction, it is has the same effect as import.meta.require in terms of downsides:

import Module from 'module'; // web is missing feature here, error

// could also just assign this to a variable, but it ends up the same as import.meta.require
import.meta.require = Module.createRequireFunction(import.meta.url);

It moves the web incompatibility to a module rather than all import.meta objects, which has ergonomic problems but prevents our loader from always having parts that are encouraging people to use CJS; they would need to opt into using non-web compat userland by using non-web compat core modules. The key difference in this opt-in vs using a runtime function is that the module graph would fail to load earlier and be easier to find ahead of running code.

I agree with @ljharb on what you see as controversial. I don't think it is a strong enough point to claim that surprise is controversial when documentation, reflection, error messages, etc. allow multiple paths of finding the solution quickly. It also leaves the solution to the consumer so they don't need to wait on upstream changes. If we were to take the API based approach using require, the consumer and the author would both need to change to use ESM based APIs.

bmeck on 29 Jun 2018

I think that we are in a classic max-min situation. Participants feel strongly on all sides, but all participants are motivated to make some kind of progress.

I assume that a user-space library can implement something like createRequireFunction. Is that true? If so, then maybe we can ship module support without import.meta.require or similar. I agree with @bmeck that we should not pollute the import.meta namespace with CJS-isms.

On the other hand, I think that import "cjs" is a non-starter for many in this group, because it will force everyone to use ".mjs", whether they want to or not.

We should consider whether we can move forward without either import.meta.require or import "cjs" as part of the default loading strategy.

zenparsing on 30 Jun 2018

I don’t think we can move forward without the latter.

ljharb on 30 Jun 2018

👍1

I think that we are in a classic max-min situation. Participants feel strongly on all sides, but all participants are motivated to make some kind of progress.

I completely disagree, I think the claim that web compatibility necessitates feature parity is fundamentally at odds with Node. The ability to load Node core modules and the ability to load CJS in any form is not something that can be used by modules seeking to be supported by both platforms. I think any solution to implementing ESM in Node mandates some manner of loading those exact modules that are not able to be supported by the web. This is the root of our controversy and it is unable to be non-controversial if people claim that the implementation of ESM in Node must require that all modules written for the Node implementation be web compatible due to feature parity. As such, I strongly believe minmax as a strategy is at odds with the concepts of implementing ESM in both platforms. If you can explain how to apply this strategy in a way that is able to remove the fundamental conflict of loading non-web features in a web platform compatible manner, I would be able to suggest minmax; however, the argument of feature parity being required for features unable to be supported on both platforms invalidates even the concept of such a solution.

I assume that a user-space library can implement something like createRequireFunction. Is that true? If so, then maybe we can ship module support without import.meta.require or similar. I agree with @bmeck that we should not pollute the import.meta namespace with CJS-isms.

Doing so once again requires usage of non-web features. Namely allowing the ability to interact with the require.cache mechanisms and/or loading the Node core 'module' library. If you can explain how this once again allows web compatibility by requiring web incompatible APIs I would be interested, but they are fundamentally at odds.

On the other hand, I think that import "cjs" is a non-starter for many in this group, because it will force everyone to use ".mjs", whether they want to or not.

The ability to import "cjs" mandating .mjs is an interesting claim. there are many proofs of allowing the ability to import CJS without requiring .mjs for all modules, I will list some here:

Your comment does not related to topic of this issue of web compatibility, which is what this issue is seeking to discuss. If there is a claim that shows how loading CJS/node builtins in any way requires module graphs not using the feature of being able to CJS/node builtins unable to load on the web I would be happy to discuss that here; however, I would prefer to keep this topic about web compatibility.

We should consider whether we can move forward without either import.meta.require or import "cjs" as part of the default loading strategy.

I agree on import.meta.require being fundamentally at odds with the web platform due to the API it uses. However, I have yet to see evidence of how the ability to load CJS mandates that modules using that feature in Node be incompatible with the web platform. We have even discussed migrations and the ability to ship multiple implementations of dependencies. I want to broaden this from import "cjs" to import "dependency" based upon the fact that the resolved module record can be multiple formats and multiple distributions for differing deployments.

Is there any proof that the ability to import "dependency" of any format for "dependency" is only possible if we do not support loading CJS using import?

bmeck on 30 Jun 2018

Sorry, I think I'm not doing a terribly good job of communicating my thoughts. I'll take some time and respond more carefully.

zenparsing on 30 Jun 2018

The ability to import "cjs" mandating .mjs is an interesting claim.

@bmeck If I could speak for @zenparsing, I assumed he was referring to the discussion above where we were discussing file-level imports, where you would need .mjs in order to do import './file' and have it resolve to either file.mjs or file.js. Of course, unambiguous syntax or "use module" would also allow for file-level import interop, but I was under the impression that those solutions were off the table. Are they?

Another thought is that we could treat “the way browsers do things,” for things that Node also does, the same way we treat spec compliance: a requirement that gets relaxed only by opt-in loaders.

GeoffreyBooth on 30 Jun 2018

@GeoffreyBooth

Another thought is that we could treat “the way browsers do things,” for things that Node also does, the same way we treat spec compliance: a requirement that gets relaxed only by opt-in loaders.

I'm not sure what this means in the context of any of this conversation. What is the intersect of things browsers do and things that Node does being applied to? Certainly loading node core libraries and loading CJS using either import or require are not in that intersection.

bmeck on 30 Jun 2018

What is the intersect of things browsers do and things that Node does being applied to?

The discussion above about import name resolution, for one: if browsers can only resolve relative or absolute URLs, that _could_ be all that Node does, too (in ESM). Support for other types of resolution could get added via loaders, just as others have proposed using loaders for some or all CommonJS interoperability.

And so on for other things that both Node and browsers support. I think the import statement is one of those things. It’s too narrow to define it only by how it’s used, e.g. “an import of CommonJS” or whatever. It’s also just syntax, and if there’s a line of code that’s an import statement that can be parsed in one environment and not in the other, like import './file.js', then that’s an inequivalence between environments. Maybe the spec permits it, but if browsers have standardized on what they allow, then that’s a _standard_ if not an official spec.

The logical conclusion of this, at least for import statements, would be a Node import so limited as to be almost impractical: basically only relative paths to files with full filenames, and those files must be ESM. (Obviously things get better once package name maps are available.) So yes, I understand why people find this unacceptable, and I agree with that assessment. But I think what the pluggable loaders proposal was getting at was, Node core/default could really be this minimal, almost unusable, and then all the nice friendly UX we want gets layered in via loaders. Then we get the best of all worlds: spec compliance and browser standards equivalence (by default), with convenience opted-into. Now, telling users to load a dozen loaders might itself be impractical, and we might not want the Node runtime to start to resemble Babel with its dozens of plugins and presets for groups of plugins, so maybe the lots-of-loaders approach is itself bad UX. I’m not sure if there _is_ a good solution here.

GeoffreyBooth on 30 Jun 2018

@GeoffreyBooth that only covers the path resolution concerns.

If we intend to also limit to supported protocols of the browser, we will not be able to load off of file: URLs.

If we intend to also limit supported modules to only modules that browsers can load, we cannot load any Node core libraries.

If we intend to also limit supported modules to only the supported MIMEs of browsers (ESM and WASM) it does not cover forwards concerns like when browsers start supporting HTML modules the fact that Node once again goes out of sync with it.

If we intend to also match cache behavior we will break existing behaviors around how indirection works for symlinks and parts of the CommonJS ecosystem.

If we intend to also match WHATWG import.meta.url behavior it will be based off the request URL instead of the response URL.

I seriously feel that a lot of these discussions are proxy discussions about .mjs and people are willing to make claims that Node should be unusable by default as you state in the face of any solution that may even consider using .mjs. If you are serious about only supporting how browsers load Modules you need to think about far more than how a specifier resolves to a URL.

Our goal should not be to make an unusable solution.

bmeck on 30 Jun 2018

@bmeck The second half of my post described how I agreed that we need to support all those things, and that there’s a school of thought that says to support them all through loaders. That might not be loaders as loaders are currently designed, it might be a radical expansion of what loaders are and can be, but there’s at least some logic to that thinking. Again, I’m not saying I agree with it. I’m also not saying I agree with locking down Node to the limited ESM functionality that browsers have. My point was that if we have to support all this extra stuff that might not be compatible with browsers, and it’s part of Node by default, we run the risk of violating our “browser equivalence” goal, which is important to many people.

For example, how do we decide when something that’s not browser-compatible is okay to include versus something that’s not? Why is import './file' okay, but import.meta.require is not?

I think it’s worth trying to define what we mean by browser equivalence, since as you write there is a long list of things related to ES modules that Node can and will do that browsers can’t. I was interpreting the “browser equivalence” goal to mean that code written for one environment executes identically in the other environment, for code that can be cross-platform. @SMotaal has done some thinking about this in #136 too.

As for .mjs, it is a part of this, sure, but it just makes for a easy-to-grasp example with regard to resolution. I’m not bringing this up as a way to try to prove that .mjs doesn’t work, if that’s what you’re thinking. Though I _do_ think there’s a problem when the exact same code _executes differently_ in browsers and Node, such as import './script.js', where script.js is an ambiguous file that could be parsed as either ESM or CommonJS. Node in --experimental-modules will treat script.js as CommonJS and parse it as Script, while browsers will treat it as ESM. (I think? Please correct me if I’m wrong.) My point with bringing up .mjs is that Node (currently, in --experimental-modules) requires import statements of ESM files to carry the .mjs extension, and browsers don’t, and that’s a potential incompatibility. I thought that this kind of thing was what the “browser equivalence” goal was meant to try to avoid. I’m not trying to make a case that we can’t have .mjs in general or that we shouldn’t use .mjs, but just that using it as part of overloading import can perhaps lead to problems. Again, please feel free to correct me if I’m wrong about any of this, I’m here to learn.

GeoffreyBooth on 30 Jun 2018

@GeoffreyBooth

For example, how do we decide when something that’s not browser-compatible is okay to include versus something that’s not? Why is import './file' okay, but import.meta.require is not?

As has been stated above, nothing about the file

import './file';

Is unable to be interpreted with browsers, and even a reference server for it is shown above. If you can show me how that specific source text is unable to be loaded in a browser we could claim it to be incompatible. On the other hand:

import.meta.require('./file')

will always fail in browsers as import.meta.require is not a function provided by browsers.

I think it’s worth trying to define what we mean by browser equivalence, since as you write there is a long list of things related to ES modules that Node can and will do that browsers can’t. I was interpreting the “browser equivalence” goal to mean that code written for one environment executes identically in the other environment, for code that can be cross-platform. @SMotaal has done some thinking about this in #136 too.

We want to support browser workflows, and the ability to author code that works in both environments. We have to my knowledge never wanted to prevent the ability to write code that does not run in browser environments. The statements that seem to keep coming up above seem to imply not equivalence, but a limited subset of features such that code cannot be written in such a manner that a browser cannot load it. I will claim this is not possible. The mere fact that we diverge from browsers on aspects is proof of that such as loading from file: URLs. The fact that we allow loaders is another. I agree that we need to define what we see as web incompatibility, but I see no way in which we are seeking equivalence.

Though I do think there’s a problem when the exact same code executes differently in browsers and Node, such as import './script.js', where script.js is an ambiguous file that could be parsed as either ESM or CommonJS.

Lets assume that it does end up pointing to the same resource in both resolvers. We need to be very clear import './script.js' executes in an equivalent manner in both environments. It is only the resource located at the resolved path for './script.js' that differs in a significant way. Nothing about import has an affect on that, it is only the environment's determining how to interpret script.js that differs. That has nothing to do with the algorithm used to locate it, and nothing to do with what the platform used to identify the MIME of the resource it loaded. It is only once we have already resolved the resource, and already determined the MIME that it differs.

Node in --experimental-modules will treat script.js as CommonJS and parse it as Script, while browsers will treat it as ESM. (I think? Please correct me if I’m wrong.)

Browsers will defer to the server on how to interpret this script.js resource. They do not care about file extension when determining MIME. It could even be a WASM module.

My point with bringing up .mjs is that Node (currently, in --experimental-modules) requires import statements of ESM files to carry the .mjs extension, and browsers don’t, and that’s a potential incompatibility.

If you can show me how requiring .mjs extension in node prevents people from writing ESM for the browser, I would appreciate it.

I thought that this kind of thing was what the “browser equivalence” goal was meant to try to avoid. I’m not trying to make a case that we can’t have .mjs in general or that we shouldn’t use .mjs, but just that using it as part of overloading import can perhaps lead to problems. Again, please feel free to correct me if I’m wrong about any of this, I’m here to learn.

I think this is where we strongly disagree. I see no path to create a system that is unable to write code that cannot be evaluated in the browser and it seems you are trying to treat that as equivalence. Even if we use userland loaders in order to mitigate this issue, those loaders themselves must use incompatible APIs.

Incompatibility is indeed about when something functions differently between systems. However, that is innate to this problem space and the reason I don't see @zenparsing 's suggestion minmax as even viable. I see people going to great lengths for this idea of browser equivalance, but at the same time casting it aside when convenient. The word "equivalance" is being misused here to mean a constrained subset of features that do not allow code to be written in a way that only runs in Node.

I opened this issue to discuss what we see as web incompatibility. Instead we are moving towards arguing that Node should not do anything outside of a specific constrained subset of browser features and calling "equivalance" even if it does not have all the features of a browser.

bmeck on 30 Jun 2018

https://github.com/GeoffreyBooth/browser-equivalence-edge-case

GeoffreyBooth on 30 Jun 2018

@GeoffreyBooth your repo is a perfect example of why .mjs is needed, and shows how browsers’ choice of disambiguation causes the ambiguity. node has different constraints, and simply can’t make the same choice - so in node, the ambiguity wouldn’t exist, and the deviation from browsers is both unavoidable and desirable.

ljharb on 30 Jun 2018

👍1

@GeoffreyBooth and Node can't even load it over HTTP, you are trying to make a claim that Node needs to create an environment where any module that loads in the Node default implementation must load in browsers in order to have your equivalence, but no such module exists. The only example that could exist would have been if Node was loading data: URLs which wouldn't have the protocol mismatch problem.

What is the exact set of features you require for your definition of equivalence, because it is not any guarantee that your module will work in a browser if it works in Node. Be it from using the wrong protocol or any of the reasons listed above. Without a definition it seems you are just throwing up examples of where you can create code that runs improperly if you don't disambiguate .js which just furthers that .js is a bad format to write browser compatible code with due to containing CJS, Scripts, and potentially ESM. You see your example as an argument on why loading .js must be ESM, but at the same time you can show a problem of loading Script via <script src=text/javascript src=/ambiguous.js>. So, if the argument is that we should not have any ambiguous cases, we have one even on the web platform without Node getting involved. Node occupying the same place of ambiguity is actually a good thing, because the users of the web platform already must be careful about .js files [due to Script and Module alone].

bmeck on 30 Jun 2018

This isn't really about browser equivalence; we definitely need Node's default loader to do something different than the browser. It's about creating a solution that is acceptable to all sides. Minimal, extensible solutions can help. We can use the limitations of the browser loader as guidance.

One of the clear requirements we're hearing both here and from the community is that users are able to use ".js" for ESM if they choose. Treating:

import "./file.js";

as CommonJS in the default loader fails that requirement.

I think that a minimal solution similar to what is described here, extensible via loaders, would work out pretty well, but I need to spend more time with it.

zenparsing on 30 Jun 2018

@zenparsing "if they choose" can be a flag or an option in a package map... there's nothing stopping cjs as the default behaviour there.

devsnek on 30 Jun 2018

can be a flag or an option in a package map

Can you provide a more concrete example of how that works? Let's say I publish a library where I use ".js" for ESM. How does my choice get preserved?

zenparsing on 30 Jun 2018

@zenparsing maybe there's some loader or package map or whatever. the point is that he most sensible default will reflect the fact that there's a lot of cjs out there, all with a js extension as the most defining feature of it and 2) there is no node-comparable (none at all!) esm out there because we're still sitting around here defining it. given those two bits of information and that "js" is a great marker of cjs for the node ecosystem, it makes a lot of sense to use js/mjs as the default

Let's say I publish a library where I use ".js" for ESM. How does my choice get preserved?

like above, out-of-band override mechanisms. assuming we don't have these mechanisms, why would you do this? why would we complicate things by supporting this? what would we gain by doing this except not having mjs? why do we need to not use mjs? is there evidence that mjs is going to destroy the ecosystem or go back in time and kill sarah connor? mjs is pretty much through ietf, google has recommended that people use it (and people will do whatever google says), and about three years of discussions have led up to it.

devsnek on 30 Jun 2018

One of the clear requirements we're hearing both here and from the community is that users are able to use ".js" for ESM if they choose.

This is separate from web compatibility, feel free to bring it up in a different issue unless it has a concern related to being unable to create modules for both platforms.

bmeck on 30 Jun 2018

👍1

So over in my neck of the woods at CoffeeScript, people are often asking for new features that require new syntax. And pretty much since 2015 the answer has always been, well, _maybe,_ if we think it’s really really safe that the ECMA committee won’t claim that syntax in JavaScript for something else. The biggest example of something like that happening is when JavaScript adopted CoffeeScript’s function argument default values, e.g. fn = function(a = 1) { return a; }, but deviated from CoffeeScript 1’s definition. In CoffeeScript 1, this example’s fn(null) returns 1, whereas in ES2015 and CoffeeScript 2, it returns null. This pissed off a lot of CoffeeScript users.

How browsers treat import './file.js' feels very similar to me. Yeah, it’s awfully annoying, isn’t it, that browsers chose to allow ESM from a file with a .js extension! Or as @bmeck would be quick to correct, browsers technically allow ESM from a URL served with a MIME type of text/javascript; but MIME types are standardized, and pretty much all servers serve .js files with MIME types of text/javascript or application/javascript, so that’s functionally the same thing as allowing ESM from .js. If all servers serve .js files with a MIME type that browsers treat as JavaScript, and all browsers treat import statements of those files as importing JavaScript in ESM mode, then that’s the standard.

There’s really a straightforward solution to the standards deviation shown in the demo: Node could treat all .js files in import statements as ESM, just like browsers do. All this means is that we need another way to import a CommonJS file. One browser-compatibility-safe way would be something like:

import { require } from 'module';

require('./ambiguous.js');

This doesn’t run in a browser, of course, but browsers also can’t import non-ESM JavaScript into an ES module. The point is that this version isn’t overloading syntax that browsers _do_ support, namely, import statements. If we want that overloading, and I’m not arguing that we shouldn’t, it can be enabled via a loader.

All .mjs is doing for us in statements like import './ambiguous.js' is enabling _transparent_ file imports of CommonJS. If we want to follow the browser standard, Node would simply not be able to do such imports transparently by default, and it would need to be achieved either through some non-transparent syntax like my example above or through a loader which overrides Node’s default behavior.

GeoffreyBooth on 30 Jun 2018

👍2

That simply doesn’t work well in node - some files are c++ (.node), some json, some will be wasm, etc. Browsers similarly won’t be differentiating by extension - i presume they’ll differentiate by mime type. Node has extensions to convey that information, since there’s no http server.

ljharb on 30 Jun 2018

MIME types are set by web servers based on extensions. There’s no difference between Node loading .mjs as a JavaScript module in ESM mode based on the extension .mjs, and a browser loading an URL ending in .mjs as a JavaScript module in ESM mode because the webserver served it as text/javascript because it had the extension .mjs.

Webservers wouldn’t serve .node files with a text/javascript MIME type, so that doesn’t matter. Just as browsers don’t treat _anything_ in an import statement as JavaScript, neither would Node. I assume browsers would process import './file.wasm' as WASM, based on the MIME type which was set by the server based on the .wasm extension; Node would do the same, albeit without the “convert extension to MIME type” intermediate step.

GeoffreyBooth on 30 Jun 2018

@ljharb To clarify, the suggestion is that node's default loader would use the extensions ".js/.mjs", ".node", ".json", ".wasm", etc. to know what to do, but import "./x.js" would import ESM.

If you want import "./x.js" to perform CommonJS-to-ESM transformation, you'd have to opt-in to that.

We'll need to prove out that this will work, of course. Given tools like "esm" however, I'm pretty confident.

zenparsing on 30 Jun 2018

mjs is pretty much through ietf, google has recommended that people use it (and people will do whatever google says), and about three years of discussions have led up to it.

Perhaps due to a personality quirk, I don't find those considerations convincing in the least. Also, a lot has changed in these 3 years. As far as I can tell, the community has more or less figured this out for us: they are authoring in ESM, dual publishing, using ".js", and using package.json:module.

zenparsing on 30 Jun 2018

👍1

MIME types are set by web servers based on extensions. There’s no difference between Node loading .mjs as a JavaScript module in ESM mode based on the extension .mjs, and a browser loading an URL ending in .mjs as a JavaScript module in ESM mode because the webserver served it as text/javascript because it had the extension .mjs.

The same is true for loading .js files as application/node which is also an officially recognized MIME for .js. You can't try to occupy both sides of this argument when .js has multiple MIMEs that are proof of it being a poor choice for anything seeking to unambiguous.

Webservers wouldn’t serve .node files with a text/javascript MIME type, so that doesn’t matter. Just as browsers don’t treat anything in an import statement as JavaScript, neither would Node. I assume browsers would process import './file.wasm' as WASM, based on the MIME type which was set by the server based on the .wasm extension; Node would do the same, albeit without the “convert extension to MIME type” intermediate step.

This gets more complicated as we scale JS to more goals.

REPL
MultiModule
Strong Mode recently

If we don't disambiguate things, the situation increases in complexity and problem space. We need to think ahead.

If we follow your arguments of not wanting to accidentally load files in the wrong format; even if we look behind using .js has problems. You don't want people to load existing .js files that are CJS (or Script) as if they were ESM in the same way that you don't want ESM to be treated as CJS or Script, or REPL, or MultiModule, or any number of formats that could be argued should be in .js files.

In both directions future and past we don't want to use .js as it remains ambiguous and only becomes more so.

bmeck on 30 Jun 2018

👍2 👎1

This gets more complicated as we scale JS to more goals.

I don’t see why this is an argument that we should deviate from how browsers work. Browsers import anything with a JavaScript MIME type as ESM. Node should import any file with a file extension that maps to a JavaScript MIME type as ESM. It’s that simple.

Node has to deal with other types of imports that browsers don’t. Fine. That doesn’t mean we need to overload the import statement to do so. We could create import { require } from 'module' for CommonJS, import { importMultiModule } from 'module' for multi-module, etc. I’m sure others can suggest other solutions. The point is, there are ways we can solve this “other goals” problem that don’t rely on file extensions and/or overloading import, and therefore don’t create incompatibilities with browsers. Just as we don’t break spec even though it would sometimes be convenient, we shouldn’t deviate from browser standards even though it would be convenient here. We can certainly solve this “how to import other goals or other types of files” problem in several browser-compatible ways. And if users want the convenience of an overloaded import statement, let them add a loader to enable that.

GeoffreyBooth on 1 Jul 2018

👍2

.js is also associated with application/node according to IANA and we are working on node, not browsers. meanwhile mjs only maps to proper JavaScript mimetypes.

devsnek on 1 Jul 2018

.js means CJS in node, and realistically always will, no matter what decisions we make and what other things it might mean.

ljharb on 1 Jul 2018

.js is also associated with application/node according to IANA and we are Node.js…

.js means CJS in node, and realistically always will, no matter what decisions we make and what _other_ things it might mean.

Yes. So name your ESM files with .mjs. Nothing about what I’ve said above says you can’t or shouldn’t.

I did a little test, along the lines of what @bmeck did above. I used this gist to run a local webserver in the repo of my demo, and I edited line 11 to give .js a MIME type of application/node, which means CommonJS according to here (and written by Bradley!). When the code in index.html tries to import './ambiguous.js', which is now served with a Content-Type header of application/node, Chrome throws an error:

Failed to load module script: The server responded with a non-JavaScript MIME type of 
"application/node". Strict MIME type checking is enforced for module scripts per HTML spec.

_This_ would be the behavior that is equivalent to what experimental-modules is doing with import './file.js'—treating .js as application/node/CommonJS, and loading it as such. But even though Bradley managed to get that alternative MIME type accepted by a standards body, there’s probably no webserver in the world that serves .js files with the MIME type application/node. For all practical purposes, servers use text/javascript or application/javascript as the MIME types for .js, both of which browsers interpret as ESM. If Node wants to treat .js as application/node/CommonJS, it deviates from how browsers and the Web do things. Arguing that because .js _can_ be served as application/node, and therefore the experimental-modules behavior of loading .js as CommonJS is spec-compliant, is facetious. Since the real world probably universally serves .js as text/javascript or application/javascript, Node would be the outlier here, doing things differently than browsers and webservers.

I really don’t understand the strong resistance to conforming to browser standards here. Importing individual CommonJS files into ESM is not one of our biggest use cases; importing CommonJS _packages_ is. If you need to use something other than the import statement to pull in a CommonJS .js file, or add a loader to overload import to enable that behavior, that’s not the end of the world. I think it’s more important that we uphold our goal of ensuring that code written to run in browsers or Node runs identically in both.

GeoffreyBooth on 1 Jul 2018

❤2 👍1

@GeoffreyBooth

I really don’t understand the strong resistance to conforming to browser standards here.

The issue here is that browsers do not have a legacy module system. For each entry point, it is either a self-contained script or a dependency graph containing only ESM. The push-back against conforming to browsers is in part due to the following identified features:

Mixed module types within app/module; gradual migration from CommonJS to ESM (#99)
Transparent interoperability for ESM importing CommonJS (#100)
Transparent migration

Node has a legacy module system, so if you understand "transparent interop" as implying "agnostic consumer", you should support import "cjs". The resistance comes from people who consider this feature to be more important than out-of-the-box browser compat.

Overall, I feel that both sides exposed their arguments but you're now talking past each other because you prioritize different features.

From what I've read here for example, I see two main approaches:

Node's ESM modules should be immediately compatible with browsers with a dumb server: explicit extensions, relative paths only, etc. Use tooling (ex: loaders) to make it work similarly to Node's CJS.
Node's ESM should remain extension-less (close to Node's CJS). Use tooling to make it work in a browser: smart server (append an extension to the requests when resolving the file on-disk) or generate package maps.

It's good to hear the technical arguments, but there's a deeper issue around which features to prioritize.

My personal opinion is that ESM will reduce the complexity of using modules in the browsers, but it won't remove the need for tooling in the browser (generate package maps, bundle for prod). In this context, I don't believe in "works out-of-the-box in browsers and Node" beyond projects with a few files. Given that, I'd prefer to avoid removing features such as import "cjs" because of the "browser compat" feature.

demurgos on 1 Jul 2018

👍2

Given that, I'd prefer to avoid removing features such as import "cjs" because of the "browser compat" feature.

Sure, but that's just one aspect. There's also the fact that import "cjs" in general gives a fundamentally broken user experience because of the lack of named imports. And the fact that import "./cjs.js" effectively (but unnecessarily) mandates ".mjs".

I agree that we've probably exhausted the conversation at this point. I think those of us that want to explore a minmax solution should continue work on a POC.

zenparsing on 1 Jul 2018

I don’t agree that it’s fundamentally broken. It’s just surprising for babel users - who browser folks, and browser compat/“no build process” advocates, keep reminding us are not the majority of javascript developers - nor even the majority of node developers, i suspect.

ljharb on 1 Jul 2018

👍1

I don’t agree that it’s fundamentally broken. It’s just surprising for babel users

That's fair, I'll take surprising :)

zenparsing on 1 Jul 2018

@demurgos

Node's ESM modules should be immediately compatible with browsers with a dumb server: explicit extensions, relative paths only, etc. Use tooling (ex: loaders) to make it work similarly to Node's CJS.

My main complaint with this presentation of one side of the argument is that it isn't encompassing all of the compatibility problems. Often people discussing web compatibility focus on one specific issue and I think it is because they are seeking a specific outcome. In particular, they focus on MIME DBs used in file servers, under the assumption that MIME DBs are non-configurable by default. They do not focus on browser compatibility but often phrase their argument as if it were. They could phrase it as web platform / ecosystem compatibility, but they do not and yet they use a basis for their main point of contention that is unrelated to browsers.

There are tons of topics on web compatibility and yet they often cast them aside seemingly for some arbitrary subset of compatibility, I can quickly point out a few that are being ignored in what I think of as a sort of tunnel vision to reach a goal even if it makes Node itself unusable:

The fact that .js CJS files uploaded to these servers in question already serve the file with the wrong MIME and how changing Node's interpretation of .js to ESM would continue to serve such files incorrectly if you just use one of the file servers that are being used as basis for their argument.
The cache handling mismatch of browsers and CJS/npm regarding indirection
The import.meta.url handling mismatch of browsers and CJS/npm
That browsers are not going to ship Loaders, and the affect of only having that feature within Node if Node itself is unusable to the point of requiring them.
The lack of http:/https:/data:/blob: URL support in Node.
The usage of Service Workers instead of Loaders and the limits/implications thereof
The problems that continue if we treat a .js file such as the 100s of thousands on npm that are CJS accidentally as ESM.

If the arguments were more universal about web compatibility I would agree with your point if you changed from "browser" to "web platform", but I don't see the overall compatibility being discussed here. I see very narrow focus and acting like it is towards universal compatibility.

If we want web compatibility to be first class we need to address all of those points, especially if we are arguing that Loaders can solve any problems and will be easy enough that the default implementation of ESM in Node can be virtually unusable without them. We shouldn't take one issue of web compatibility as solving all of the others. We need to go through and discuss what is wrong with things today, what will be solved by implementing a solution, and what will not be solved or potentially broken by implementing that same solution.

I once again, started this issue in an attempt to talk about web compatibility as a whole and a personal desire to discuss if the ability to load files with a application/node MIME is seen as web incompatible. I do see any ability to load CJS as being incompatible and am surprised at people wishing to be unable to load CJS at all in the name of web compatibility. We have 100s of thousands of modules, and some people seem all too eager to abandon all of the work of those users in the name of web compatibility but don't even consider the overall compatibility, and instead focus on one specific issue.

But even though Bradley managed to get that alternative MIME type accepted by a standards body, there’s probably no webserver in the world that serves .js files with the MIME type application/node.

It didn't need to be standards track, I wanted it to be vendored. IANA requested we resubmit it as standards tack. I'd be careful in trying to paint this effort as entirely on me.

bmeck on 2 Jul 2018

👍1

Concerning valid JavaScript import statements executing _differently_ in Node and in browsers, it’s not just the one example we’ve been discussing above, where import './file.js' pulls in a file as Script/sloppy mode in Node --experimental-modules and as module/strict mode in Chrome. There’s also:

Browsers are working toward handling bare import specifiers via package name maps, whereas I think --experimental-modules uses a Node-specific resolution algorithm (assuming that the implementation follows this plan). We could have issues similar to import './file.js' in the future, if import 'packagename/something' sometimes resolves differently in Node as browsers. Browsers following one spec while Node follows a different spec is a pretty sure recipe for incompatibility.
The cases of import './file', where ./file resolves to either file.mjs or file.js; and import './folder', where ./folder resolves to either ./folder/index.mjs or ./folder/index.js (or other possibilities?). These “add on the file extension” and “find the folder index file” cases would be compatible with browsers only if webservers behaved the way that Node’s resolution algorithm does, and pretty much no uncustomized webserver will resolve an URL ending in file to file.mjs or file.js, or an URL ending in folder to folder/index.mjs or folder/index.js.

And I’m sure there are more cases that others can think of. What it boils down to is that we’re treating the import specifier string as a blank slate that the JavaScript spec allows us to define as we please, while browsers and WHATWG treat it as something that should be standardized upon. (I encourage @bmeck to explain this far better than I can, and to please correct me.)

Part of the complication here is that browsers do whatever the webserver tells them to do: if a customized webserver resolves the MIME type for ./app.foo as application/javascript, the browser will load app.foo as a JavaScript module. And as far as I know, there’s no spec that defines webserver behavior for serving JavaScript or resolving URLs in the context of serving JavaScript. If there was such a spec (is there one?) then that could be the blueprint for what Node should do: if webservers are told to serve .js files as application/javascript, for example, then that’s a spec that Node would be expected to follow as it stands in for the webserver’s role when “serving” files as JavaScript code to be imported into the runtime. As far as I can tell, there is no such spec, and the WHATWG spec and our discussions are based on assumptions of how uncustomized traditional webservers behave.

These ambiguities or gaps between the specs let Node do custom things, like treating import './file.js' as importing a script or import './file' as importing a file named file.mjs, without technically breaking the JavaScript spec. But doing such things relies on Node behaving unlike any standard webserver, when you treat Node’s loading of files as the equivalent of a webserver serving the content of files to browsers. There’s an argument to be made that Node shouldn’t care how browsers and/or webservers behave, or that Node shouldn’t follow even the parts of the WHATWG spec that _could_ overlap with how Node does things; but simply ignoring that stuff leads to incompatibilities, where the same code could execute differently in Node as in browsers paired with real-world webservers.

I understand that this sounds awful to a lot of people, like I’m saying that we have to abandon all interoperability with CommonJS or any number of our other goals. That’s not the case. I’m saying that we need to achieve those goals _in a different way,_ other than using the import statement in ways that are incompatible with browsers and webservers. One such way would be via pluggable loaders; another would be via new custom functions that Node supplies on its core modules. I’m sure people can come up with other clever solutions. But I think it’s dangerous to conclude that our interoperability goals are more important than browser compatibility, and therefore it’s okay that the same code executes differently in the two environments. I think we can have our interoperability _and_ our compatibility, and as a standards-compliant member of the JavaScript community Node.js should produce an implementation that achieves both.

GeoffreyBooth on 2 Jul 2018

❤1

The language spec specifically indicates that import specifier are indeed such a blank slate; and browsers are standardizing on it for browsers. It is totally fine for node to have different meanings for them - especially if package name maps allows node users to configure browsers to match node’s algorithm - that’s the path we should be focusing on, imo: making browsers more like node where possible, not the reverse.

Webserver already can and do automatically resolve file extensions; this is trivial to do for php with .htaccess, for example. It is perfectly compatible with browsers even if webservers were required to do path rewriting - but with package name maps, they won’t be.

ljharb on 2 Jul 2018

👍3

The cases of import './file', where ./file resolves to either file.mjs or file.js; and import './folder', where ./folder resolves to either ./folder/index.mjs or ./folder/index.js (or other possibilities?). These “add on the file extension” and “find the folder index file” cases would be compatible with browsers only if webservers behaved the way that Node’s resolution algorithm does, and pretty much no uncustomized webserver will resolve an URL ending in file to file.mjs or file.js, or an URL ending in folder to folder/index.mjs or folder/index.js.

It actually is much worse than this. Even if you do HTTP redirects you will be generating separate module records in the client. You have to completely intercept import and redirect using a complex handshake like in https://github.com/bmeck/esm-http-server . The capabilities of the browser make it significantly hard for even a smart server to handle aliasing, which makes any form of indirection non-trivial. It also requires some tricks to manipulate the contextual data for the module like rewriting import.meta.url if you want that to match the alias destination.

Browser compatibility with any form of indirection is hard enough to the point that I question the viability of any indirection being allowed if web compatibility is a goal.

What it boils down to is that we’re treating the import specifier string as a blank slate that the JavaScript spec allows us to define as we please, while browsers and WHATWG treat it as something that should be standardized upon.

This is indeed true, but we are not beholden to treat browsers as our absolute master/slave relationship where Node is slaved to browsers. They have different constraints when shipping applications than Node does. We should not be eager to adopt those constraints. We can agree on an intersection without needing to carry the burden of taking on the entirety of compatibility concerns, but we can only do this if we realize we are not seeking true compatibility but instead a common subset that can be documented and standardized upon.

WHATWG is seeking to standardize behavior for browser vendors, it is and has largely been unconcerned with difficulties that introduces for Node in the past and is part of why I heavily believe we shouldn't treat a standard as being the standard we should use. If Node had more power in WHATWG or I had more pleasant experiences with them in the past I might be more open towards treating them as a force of good for Node, but that has not been the case and I have largely become wary of interactions with the members of WHATWG.

In particular, as @ljharb points out, the JS specification which we cannot ignore does treat specifiers as a blank slate of sorts. We can build upon that and ensure that there is a common subset of safe behavior with browsers without adopting the full compatibility matrix and essentially needing to implement all of the browser infrastructure and constraints thereof in Node.

If there was such a spec (is there one?) then that could be the blueprint for what Node would follow too: if webservers are told to serve .js files as application/javascript, for example, then that’s a spec that Node would be expected to follow as it stands in for the webserver’s role when “serving” files as JavaScript code to be imported into the runtime.

This is the purpose of the MIME registry https://www.iana.org/assignments/media-types/media-types.xhtml . WHATWG explicitly does not concern itself with file extensions because they use content-type. You may be wishing to use the WHATWG mime sniffing standard instead? They make mention of file extensions in mime sniffing not being used for sniffing mimes in any given context. In addition they have a big red box around any sort of mime sniffing in a script context and it largely is not going to help because mime sniffing is often disabled when loading scripts. You may refer to an HTML specification callout about mime databases for <input type=file> as well which is left up to hosts but is also called out as ambiguous since different platforms/hosts may return different values for different files. Overall browsers are not defining what file extensions map to nor how to sniff a MIME in a scripting context.

These ambiguities or gaps between the specs let Node do custom things, like treating import './file.js' as importing a script or import './file' as importing a file named file.mjs, without technically breaking the JavaScript spec.

There is no technicality about this topic.

But doing such things relies on Node behaving unlike any standard webserver, when you treat Node’s loading of files as the equivalent of a webserver serving the content of files to browsers.

This idea of not matching files being served from a static file server is true to the state of the world today as often application/node files would be served as text/javascript, but I am seeing little discussion to how making things act like a webserver impacts all of the existing code as it exists today.

but simply ignoring that stuff leads to incompatibilities, where the same code could execute differently in Node as in browsers paired with real-world webservers.

I agree completely, but don't see the solution to be to act like web servers. Node made what I think of as a major mistake in using .js for CJS but I think we would repeat this mistake by using .js to also mean ESM since we cannot abandon all of our legacy codebases and are not seeking to remove CJS from Node. You can use .js still in opt-in fashions like the "mode" flag" which I fully endorse using as a path to allow using .js to mean text/javascript, but the legacy is to treat .js as CJS and I see little discussion of how to handle all of the files that exist today if they are mislabelled as text/javascript instead of application/node.

I understand that this sounds awful to a lot of people, like I’m saying that we have to abandon all interoperability with CommonJS or any number of our other goals. That’s not the case. I’m saying that we need to achieve those goals in a different way, other than using the import statement in ways that are incompatible with browsers and webservers.

I don't think it is awful on premise, but I do think web compatibility is larger than just changing the file extension lookup from application/node to text/javascript.

One such way would be via pluggable loaders;

We need to be very careful here. Loaders are not coming to browsers, and they can easily be written in a way where one loader conflicts with another. Our default implementation should not rely on loaders as well because if people are encouraged to use a loader at all times, why is it opt-in. Why instead isn't the problematic behavior opt-out. Consider Strict Mode for a second. You opt-out of legacy behavior, not into it. With Modules in the JS specification you opt-in to new features by using the new grammar goal, you do not opt-out of the Script goal.

This balance of what should be opt-in and what should be opt-out is precarious. Opting out of legacy behavior and opting in to new behavior seems safer both the status quo and the way to achieve the least amount of breakage. Opting out of new behavior means that legacy codebases must change in order to run in w/e environment is now treating them as non-default behavior. That is a big change and quite a breaking one.

another would be via new custom functions that Node supplies on its core modules. I’m sure people can come up with other clever solutions. But I think it’s dangerous to conclude that our interoperability goals are more important than browser compatibility, and therefore it’s okay that the same code executes differently in the two environments.

I would note that if you serve your CJS files with application/node as you showed above and vice versa for ESM files, they would not execute differently in the different environments. They would error out in browsers before they execute and are able to run incorrectly.

I think we can have our interoperability and our compatibility, and as a standards-compliant member of the JavaScript community Node.js should produce an implementation that achieves both.

I think we all are making arguments towards this goal, just different considerations on what is important to reach that goal and what the default behavior should be.

bmeck on 2 Jul 2018

I think we've zeroed in on the primary pivot point:

Whether automatic CommonJS-to-ESM transformation should be opt-in or opt-out.
And if it's opt-in, the whether that choice applies to all module paths or only across package boundaries.

The advantage of opt-out is that we get immediate interoperability with CJS code. The disadvantages are:

If we automatically transform at the file level,
- We introduce semantics that are clearly web-incompatible and not addressed though package maps.
- We basically force ".mjs" on users, whether they like it or not.
If we automatically transform at the package level:
- The transform is lossy due to the lack of named imports; this will be surprising to consumers and will likely result in lots of bugs getting filed on packages. It won't "just work".

An opt-in approach to CommonJS-to-ESM transformation avoids those downsides, but leaves a huge open question about how the ecosystem will move forward in a world with two module systems that must be glued together by users. We need a POC that demonstrates how that will work (and I've been fiddling around with that).

There are other questions, like:

Should directory imports be supported?
Should automatic file extension searching be supported?

But I think we should stay focused on the big question above.

zenparsing on 2 Jul 2018

😕1

Whether automatic CommonJS-to-ESM transformation should be opt-in or opt-out.

I agree this is a major point, but also think this encompasses a different topic than identification of if a .js file is application/node.

I'd add a point on disambiguation:

Whether of .js files are identified as text/javascript or application/node

Even if we identify a file as application/node we could prevent creating a facade for them and error out by default.

And if it's opt-in, the whether that choice applies to all module paths or only across package boundaries.

We can narrow this on per-file as well. Likewise it would apply to disambiguating the MIME as a separate topic from the facade being provided by default or not.

We introduce semantics that are clearly web-incompatible and not addressed though package maps.

I disagree with this conclusion if we are talking about web browser, but do see ecosystem concerns both on Node's side and the Web. This applies if we are trying to match dumb static file servers as part of web compatibility but not if we solely are focused on browser compatibility.

We should try and figure out if we ate trying to match just browser compatibility and if we are also trying to match MIME DBs used by static file servers.

We also need to refine the phrase incompatible because serving files that are CJS as text/javascript remains incorrect for servers. In addition browsers won't load application/node and questioning if that is incompatible or not since it is an error in browsers.

The transform is lossy due to the lack of named imports; this will be surprising to consumers and will likely result in lots of bugs getting filed on packages. It won't "just work".

I don't understand this point. It probably needs a more lengthy explanation. I'm guessing this is an argument about the specific facade we could provide to load CJS, not the ability to load CJS? I think part of my confusion here is it is making and argument about loading unsupported specifiers but is also not addressing things like loading specifiers in the Web that are currently not supported in Node (namely data: and https: URLs).

An opt-in approach to CommonJS-to-ESM transformation avoids those downsides

It also introduces compatibility concerns with all the existing .js files that are not ESM. I'd like to see those discussed. This suggestion of pivot points also doesn't touch on any of the other points of potential compatibility issues as I point to above. We shouldn't gloss over this and state that by doing something we have no downsides. If we serve CJS as text/javascript that is still going to lead to problems.

We need to discuss the issues about why things should/could work a specific way and the implications of doing so. I am much more interested in understanding the implications of shipping an implementation than seeing a PoC that runs code in some specific fashion.

On that Note, I think if we are aiming for web compatibility we need to make it much more closely aligned with browsers than most others would probably. I would state that if we want web compatibility as the goal, we need to go to great strides around all of the points I commented on above. I don't think we should focus on just pathing concerns, or disambiguation of MIMEs.

bmeck on 2 Jul 2018

It’s not lossy because CJS only ever has a single default export. Named exports from CJS, despite babel leading people to believe they should be there, are not actually a thing that conceptually exists.

ljharb on 2 Jul 2018

It’s not lossy because CJS only ever has a single default export

It's lossy from the point of view of the user that has been writing import { x } from "lksjdf" for the past 4 years. I myself have been guilty of this. I was honestly surprised (and I was shocked that I was surprised) by the lack of support for named imports.

zenparsing on 2 Jul 2018

Yes - for a babel user. But not for a CJS user.

I do want named imports for CJS, but i think it is unproductive to characterize them as a necessity, or the lack of them as a loss. Having them is a bonus, is all.

ljharb on 2 Jul 2018

Yes - for a babel user. But not for a CJS user.

I wonder if that's a distinction without a difference in 2018?

I don't think it's unproductive to characterize the lack of named imports as a loss. Let's say that I'm starting a new project using native ESM, and let's assume that we do automatic CJS->ESM transformation on packages.

I start like this:

import foo from 'some-package';

It works, and I'm happy. Then later, maybe in another file, I do this:

import { x } from 'some-package';

I think it's going to work, because it used to in my Babel projects, or maybe because I see code on Stack Overflow that does that.

I get an error saying that 'some-package' does not have an "x" export.

This is a bit frustrating. So either I go back to the default export thing, or I leave a bug report on 'some-package' saying that their thing is broken. Either way, I feel like either 'some-package' or Node is somehow broken.

Let's restart the scenario, but this time let's assume that we don't do any automatic CJS-to-ESM transformation.

I start like this:

import foo from 'some-package';

I immediately get an error message saying "some-package does not support native ESM modules yet". I go leave a bug report on "some-package". The developers there add an ESM module entry point with all the required named exports and publish it.

Maybe the entry point looks like this:

const SomePackage = import.meta.require('./index.js');
export default SomePackage ;
export const { Component, createElement } = SomePackage ;

I update my dependency and everything works.

What happens if it's an older library that isn't supported anymore? Well, either someone decides to start supporting the package or I fall back to import.meta.require. Hopefully the former, but there's certainly a risk of the latter. (Perhaps we can smooth the transition by having npm "nudge" authors into publishing ESM entry points...)

The good thing about the second scenario is that there's no magic trying to smooth over the impedance mismatch. It's all very simple and intentional.

zenparsing on 2 Jul 2018

I'm not sure how it's different in the second scenario from "Either way, I feel like either 'some-package' or Node is somehow broken.". If you're going to assume that, you're going to assume that either way - and that means that all non-ESM packages would be instantly considered "broken" by many many people. That's a large burden to place on the ecosystem.

The other risk is that you'd read a bunch of advice telling you to "just use CJS, stop using ESM" - then ESM doesn't get adopted. The friction for people to use ESM in the first place needs to be minimal - the second scenario you outlined is much worse because there's no workaround to continue using import.

ljharb on 2 Jul 2018

I update my dependency and everything works.

And then it becomes ESM. The the library consumer must then update your dependency to the ESM package version. Since you can't have the .js file be both ESM and CJS we have to figure out a migration path and it will be breaking if anyone used deep file paths. The library consumer must change APIs to load the library from import.meta.require to import of some kind; so you break consumers anyway. This could work somewhat transparently for static imports, but might be problematic for things if you have dynamic paths such as different files for different distribution mechanisms. However, it is a breaking change migration on both the library and on the consumer.

I'm not sure that any solutions are simply going to be without downsides.

bmeck on 2 Jul 2018

👍1

I'm not sure how it's different in the second scenario

The difference, in my mind, is that instead of the situation being like "hey this should work but maybe it won't, who knows", the situation is like "here's the upgrade path, go forth and make it happen". Given a little bit of lead time, I think the ecosystem can adapt really quickly.

The other risk is that you'd read a bunch of advice telling you to "just use CJS, stop using ESM" - then ESM doesn't get adopted. The friction for people to use ESM in the first place needs to be minimal - the second scenario you outlined is much worse because there's no workaround to continue using import.

I think this point was really convincing a couple of years ago, but now I'm not so sure. ESM modules have won; I think the population of voices calling for boycott of ESM is much reduced.

zenparsing on 2 Jul 2018

I think this point was really convincing a couple of years ago, but now I'm not so sure. ESM modules have won; I think the population of voices calling for boycott of ESM is much reduced.

I don't think web compatibility is really what you are arguing when you argue about how surprises are reason to not ship support for a feature. You talk as if ESM has won, but are pointing to the fact that for years people have been using babel to compile to semantics of CJS. I don't think ESM has won so much as ESM syntax and static guarantees about their module graphs. All of those modules you are concerned with supporting are using CJS semantics, node's path resolution, and don't use import.meta.require as how to load CJS.

I think the points of evidence you are using about people using babel to the point of it being the format that "won" are at odds with stating that we don't need to support loading CJS.

bmeck on 2 Jul 2018

👍2

@bmeck With regards to your comments about having to deal with those other aspects of web compatibility: yes, those need to be discussed too. I feel like this issue is becoming a run-on thread, though, perhaps we should split these out into different discussions?

I had a few random thoughts in response to some of the above comments:

I draw a distinction between code that _doesn’t_ execute in one platform or the other, like import fs from 'fs', and code that executes in both Node and browsers but executes _differently_ between the two. The former feels like what we expect nowadays: it’s common to feature-detect, and not every JavaScript runtime supports every API, so we’re aware that we need to be conscious of our environment and what capabilities it has. The latter, though, of code executing _differently_ between standards-compliant browsers and Node, feels very much like a spec violation, like what we used to see in the bad old days when browsers would add new functionality but certain things would behave differently based on the browser. If you’re looking for where to draw the line between what web compatibility is reasonable for Node to support, avoiding identical code executing differently is a good place to start.
Speaking of package name maps and the issue of importing .js files as ESM in browsers, what happens when a package name map maps a specifier to a .js file? Node would load it as CommonJS, but browsers would load it as ESM (or try to, and fail)?
When I asked why import './file' is okay (where file is intended to be resolved to file.mjs) while import.meta.require is not, what I meant was: if import.meta.require is unacceptable because it doesn’t exist in browsers and will never exist, why is automatic file extension resolution acceptable when it also doesn’t exist in browsers and will never exist there? I would argue the the automatic file extension resolution is _worse,_ because at least with import.meta.require you’re explicitly using an API that doesn’t exist at all in one platform, whereas the import statement exists in both but can only accept certain types of input in one or the other.

To try to answer my own question, I would surmise that @ljharb would respond that import './file' is okay because a) we need a way to import CommonJS or ESM, if file could be resolved into either file.mjs or file.js; b) the spec lets us do it; c) it’s traditional in Node to leave off file extensions. (Of course, @ljharb, please feel free to speak for yourself and reject my awful mischaracterizations 😄)

To which I would reply: a) yes we need a way to import either one, but that doesn’t mean we need _this_ way; we could just as easily leave import for specifiers that are browser-compatible and use something like import { require } from 'module' (or import.meta.require) for importing CommonJS; b) the spec does allow it, but probably shouldn’t, because browsers and Node should each be treated as first-class citizens and should be standardizing on a common spec for this, and only seem to not have done so because of dysfunction between TC39 and WHATWG (per @bmeck above) and sure it sucks to be a slave to browsers especially when they made the wrong decisions, but in this case it makes sense to follow their lead because they actually have a spec for import specifiers while the TC39 spec is more or less silent on the matter; c) leaving out file extensions may be idiomatic for CommonJS, but it’s _not_ traditional in ESM (yet) and isn’t even allowed in browsers, in any practical sense (excluding unrealistic webservers explicitly configured for this purpose).

It seems to me like the real objection here is that import.meta.require or import { require } from 'module' aren’t transparent, and some folks really want a transparent CommonJS import from an import statement, even if that’s impossible in browsers. Is that the case? Because basically where I come down on this is that there’s no possible way to make an import statement work in a transparent fashion for CommonJS: you don’t get static imports, and you get incompatibilities with browsers. And if we can’t make import work to transparently import CommonJS, we might as well create a fully separate interface for that interoperability, and then there are no compatibility issues now or in the future.
@bmeck, how do you feel about import { require } from 'module', or from some other core module, as opposed to import.meta.require? I understand the objection to not wanting to pollute the namespace of import.meta, but what about attaching require to a core module?

GeoffreyBooth on 3 Jul 2018

@GeoffreyBooth package name maps would presumably resolve to a final path - no extension lookup. Thus, import paths would continue to be best extensionless - node's default implementation would do the lookup, and the package name map used in the browser (or even if one was generated and used in node) would handle any possible extension lookup. I don't see a conflict say.

I don't think it's unrealistic for a webserver to be configured to automatically add a .mjs or .js extension based on the request mime type.

import { require } from 'module' simply won't work, because it can't provide contextual information about the importer - specifically, it has to be syntax, or CJS's module-specific injected variable, to be able to do things like path resolution relative to the current module.

ljharb on 3 Jul 2018

👍1

package name maps would presumably resolve to a final path - no extension lookup. Thus, import paths would continue to be best extensionless - node’s default implementation would do the lookup, and the package name map used in the browser (or even if one was generated and used in node) would handle any possible extension lookup.

I don’t quite follow this. Are you saying that Node wouldn’t be using the package name maps? It would just do its own, different package name resolution?

I don’t think it’s unrealistic for a webserver to be configured to automatically add a .mjs or .js extension based on the request mime type.

It’s not unreasonable for a webserver to be _configurable_ to be able to do this, just as most servers can be configured to redirect http to https, for example. But many users lack full control over their webservers, and certainly over their CDNs or other things like S3 static file hosting. And just because it’s theoretically possible to create a webserver that emulates Node’s behavior doesn’t strike me as a persuasive argument for why Node should do something that no real world current webserver does.

import { require } from 'module' simply won’t work, because it can’t provide contextual information about the importer - specifically, it has to be syntax, or CJS’s module-specific injected variable, to be able to do things like path resolution relative to the current module.

I think you’re saying that a function that takes only a string as input lacks all the information it would need to behave like CommonJS require? As in, the function wouldn’t know the context from whence it was called, and therefore anything about the module doing the importing, which it would need to know in order to resolve a relative path? And I guess import.meta has that information?

Fair enough. Why not just create a new function then?

import { importCommonJs } from 'module';
importCommonJs(import.meta, './some-commonjs-file.js');

I’m sure others can come up with prettier solutions; the point isn’t to bikeshed, but to demonstrate that there are ways other than overloading the import statement to accomplish importing CommonJS files and packages.

Once again, I don’t see why it’s better to overload the import statement, creating incompatibilities with browsers, when something else could be created that’s an independent API that browsers wouldn’t be able to execute (unlike some import statements that they _could_ execute but would do so differently than Node would).

GeoffreyBooth on 3 Jul 2018

Yes, I'm saying node would, by default, have its own, internal resolution - but it would defer to a package name map if one was provided.

Creating a new function with import.meta is certainly an option, but that's hardly ergonomic.

It's not overloading the import statement, nor is it creating incompatibilities - it's precisely what import is meant for (to provide host-dependent resolution), and it can be made compatible with browsers. "Compatible with browsers" does not have to mean "compatible with browsers and static webservers without a build process", especially since that's not the common case (2 out of 3 might be common, but all 3 is not).

ljharb on 3 Jul 2018

👍1

I feel like this issue is becoming a run-on thread, though, perhaps we should split these out into different discussions?

Perhaps but we are all relating these topics towards web compatibility, or at least the intent of this thread is to do so. Overall it does seem we are making some progress in identifying the root points of interest as @demurgos pointed out.

I draw a distinction between code that doesn’t execute in one platform or the other, like import fs from 'fs', and code that executes in both Node and browsers but executes differently between the two. The former feels like what we expect nowadays: it’s common to feature-detect, and not every JavaScript runtime supports every API, so we’re aware that we need to be conscious of our environment and what capabilities it has. The latter, though, of code executing differently between standards-compliant browsers and Node, feels very much like a spec violation, like what we used to see in the bad old days when browsers would add new functionality but certain things would behave differently based on the browser. If you’re looking for where to draw the line between what web compatibility is reasonable for Node to support, avoiding identical code executing differently is a good place to start.

Can you be explicit on which spec is being violated, in particular the reason it executes differently in different platforms here is because it is using a different MIME. The solution is to disambiguate the MIMEs, either by choosing to default to text/javascript or application/node. The difference is not in how code executes but in how the platform thinks the code needs to be run. This idea of allowing the host to choose the way in which code needs to be run is a fundamental design of how ESM works and is the reason for Abstract Module Records being part of the JS specification since it was first introduced.

If you want to argue that the code is identical, we would also need to argue that the MIME is identical. Your concern is valid that file servers can serve valid source text with the wrong MIME. They currently do that today with CJS. In the future they can do they would still do that with CJS under what you are proposing. Even if we disambiguate in Node, it seems like this forced mismatch of MIMEs problem would remain. If we do move to default to text/javascript in Node we would match how file servers act today, but also not provide any sort of clear signal about how it is potentially ambiguous to use .js since it has multiple ambiguous MIMEs. I think that even if we default to text/javascript we still need to disambiguate application/node. This is part of why I don't like the idea of the method of consumption being the way in which we disambiguate MIMEs. It would allow files to have multiple MIMEs depending on method of consumption.

As @ljharb pointed out, it only takes one import 'fs'; in your entire graph to make the entire graph fail. We can treat failure to link separate from feature detection, but it isn't just core modules, but any module specifier that would not work between platforms, such as file:///app/config.js or https://localhost/script.php. It also as you pointed out will fail in a successful manner for servers that correctly identify CJS as application/node.

I have a bias towards fixing both the ambiguity that is causing us problems and heading towards the ESM workflows that is so popular as @zenparsing pointed out. I don't think we can achieve a solution that is perfect and I don't think we can fix all problems, but we can certainly choose a path that allows both ergonomics and a solid path of compatibility between platforms. The big issue here is I think we have a heavy disagreement on if a file is considered identical if it is served with a different MIME than what Node uses as the MIME for the file. It seems that you treat the file as identical and I treat is as fundamentally different. I think we can continue to go round and round on this discussion on which is the correct way to define identity but it would be resolved either way if we fully disambiguate MIMEs.

When I asked why import './file' is okay (where file is intended to be resolved to file.mjs) while import.meta.require is not, what I meant was: if import.meta.require is unacceptable because it doesn’t exist in browsers and will never exist, why is automatic file extension resolution acceptable when it also doesn’t exist in browsers and will never exist there? I would argue the the automatic file extension resolution is worse, because at least with import.meta.require you’re explicitly using an API that doesn’t exist at all in one platform, whereas the import statement exists in both but can only accept certain types of input in one or the other.

This relates to your previous point about code running incorrectly. If your file is written in such a way that it does actions before using import.meta.require such as registering a custom element, it would permanently have a side effect that potentially causes errors on the page. Your code runs up until import.meta.require but it at least errors at that point and may or may not be written to be robust against that error.

It isn't that it doesn't exist, it is that it can cause code to be written that loads in both platforms and then executes differently. With specifiers/MIMEs that don't exist in either platform, the code will fail to load and thus not execute.

why is automatic file extension resolution acceptable when it also doesn’t exist in browsers and will never exist there?

In browsers, correct; but the web platform is not just browsers it must include servers in order to get access to MIMEs and load resources. Servers can solve this problem, and further under the current package-name-maps proposal bare names can map non-extensions to extensions.

I would argue the the automatic file extension resolution is worse, because at least with import.meta.require you’re explicitly using an API that doesn’t exist at all in one platform, whereas the import statement exists in both but can only accept certain types of input in one or the other.

This seems a fine argument but doesn't seem to be about loading CJS (which is always unable to be loaded on the web). It is about the path resolution algorithm we choose. In particular, the claim that import.meta.require is more compatible with the web must exclude the fact that loading application/node on the web is not possible because import.meta.require is a way to load application/node.

My concern is about the fact that it is API based partly, but also around the idea of method of consumption defines format. If method of consumption defines format, tools and servers will be unable to use any disambiguation method to find the correct MIME for a file if .js has multiple associated MIMEs (.js has a backwards compatibility constraint on CJS existing in it so it at least has application/node somehow).

Stating that the resolution algorithm doesn't match is a great argument to have though.

a) yes we need a way to import either one, but that doesn’t mean we need this way; we could just as easily leave import for specifiers that are browser-compatible and use something like import { require } from 'module' (or import.meta.require) for importing CommonJS;

Yes, there are alternatives but I still would think we must disambiguate these files in question. Method of consumption determining format permanently prevents that disambiguation and continues the problem at hand. I'm fine with alternatives if they are as suitable as using import, however I don't see good alternatives currently. Alternatives that don't match the cow path of using import to load CJS and alternatives that produce ambiguity are possible but seem problematic to myself.

the spec does allow it, but probably shouldn’t, because browsers and Node should each be treated as first-class citizens and should be standardizing on a common spec for this, and only seem to not have done so because of dysfunction between TC39 and WHATWG (per @bmeck above) and sure it sucks to be a slave to browsers especially when they made the wrong decisions, but in this case it makes sense to follow their lead because they actually have a spec for import specifiers while the TC39 spec is more or less silent on the matter;

The ESM spec was created with the intent of it being used to load non-JS based Module Records, that is the reason for Abstract Module Record. It was never intended to have a single host define a canonical list of possible Module Record Types. There is dysfunction between WHATWG and other bodies but it is not the reason for Abstract Module Record. TC39 foresaw that things like HTML/CSS/etc. being loaded via import and also had people on committee using Node that knew about loading JSON/C++/etc. The people on TC39 were looking at package.json based disambiguation for CJS as well when ES6 was ratified, but they had not done thorough research by the time it was adopted.

The point of contention here is if Node is constrained to WHATWG or to TC39. I heavy disagree on constraining our implementation to WHATWG because we have a different set of concerns and are not going to match the WHATWG spec anyway if we don't use HTTP. TC39 is the language body and is safe for us to expect platform agnostic decisions from, WHATWG is not. Also, even if we treat .js as application/node we would be compatible with WHATWG's requirements on conformance criteria so this is somewhat a moot point if we are stating that .js should be text/javascript.

leaving out file extensions may be idiomatic for CommonJS, but it’s not traditional in ESM (yet) and isn’t even allowed in browsers, in any practical sense (excluding unrealistic webservers explicitly configured for this purpose).

It is allowed in browsers (because they are only handling what servers generate), package-name-maps also allow a form of this, and it is somewhat common in Babel modules. As has been pointed out what people think of as ESM is traditionally Babel modules, so I'm not sure what the statement about it not being traditional is referring to as the basis for this.

It seems to me like the real objection here is that import.meta.require or import { require } from 'module' aren’t transparent, and some folks really want a transparent CommonJS import from an import statement, even if that’s impossible in browsers. Is that the case? Because basically where I come down on this is that there’s no possible way to make an import statement work in a transparent fashion for CommonJS: you don’t get static imports, and you get incompatibilities with browsers. And if we can’t make import work to transparently import CommonJS, we might as well create a fully separate interface for that interoperability, and then there are no compatibility issues now or in the future.

I don't think "transparent" is a good word as I've said before. My objection is more rooted in expected workflows, migration paths, removing ambiguity of format, and allowing exclusion of CJS API usage for multiple platform module graphs. It being done using import to load CJS is somewhat secondary to me at least (though certainly heavily influenced by them). import was designed to allow this usage and I would prefer we unify on one standard way to load resources if possible, so it seems obvious to align on. I am not convinced that import.meta.require gives me any of those things listed. import certainly has some surprises regarding things like the list of named imports not being dynamically generated at runtime, but that is part of ESM and a mismatch with Babel modules and seems inevitable. I don't think something as small as named imports being a surprise affects me since we can just destructure or use property access instead. I am concerned about features and workflows, not that we use a specific API so much.

You say you get incompatibilities with browsers, but you are only pointing towards incompatibilities with serving CJS files with the wrong MIME which is solely on the server.

then there are no compatibility issues now or in the future.

This is simply not true, as I stated above. We shouldn't claim that any solution has no downsides. See the list of concerns I have with import.meta.require here and elsewhere for example.

bmeck on 3 Jul 2018

❤1 👍1

Can you be explicit on which spec is being violated

The whole idea of import statements loading .js files as CommonJS and .mjs files as ESM, and this being both spec compliant and compatible with browsers, rests on this justification:

There is no spec that governs how webservers should set the MIME types for the files they load. We see this in how most webservers are configurable in terms of what MIME types to use for each file extension, like Apache’s AddType application/javascript .mjs.
Therefore, Node is unconstrained in how it emulates the webserver’s part in serving a file for browser runtimes to evaluate, where the webserver sets the MIME type for the browser to read. Node can decide to do the equivalent of setting the MIME type for .js files as application/node (CommonJS), and this is spec compliant.

I think this logic is flawed. Just because there’s no official spec that governs what MIME type webservers should use for .js, it’s not like there’s any doubt what the _de facto_ standard is: text/javascript or application/javascript, which are interchangeable (the former is deprecated by the latter, according to the MIME spec). There’s no chance that webservers will change from this behavior, since switching .js to application/node would be a breaking change (any ESM importing .js in browsers would break) and there’s no benefit to webservers or browsers in using a MIME type that no browsers recognize or are ever likely to recognize.

So Node treating .js as application/node goes against this pretty universal de facto standard across probably all real world webservers. And there’s a real consequence, as browsers will load .js files differently than Node will. That, to me, is a spec violation.

This idea of allowing the host to choose the way in which code needs to be run is a fundamental design of how ESM works

And I think this idea was mistaken. Many users don’t have the control over their hosts to configure how they serve files; especially for JavaScript, which is a static asset often served via CDN.

Let’s shift perspective for a second, to that of a package author. Say I’m a package author, of a package such as, say, jQuery. My package is only meant for browser usage, and I don’t care about Node. I want my package to be usable from an import statement in browsers that support ESM. Do I publish my package with a root file of jquery.js or jquery.mjs?

The answer is clear: .js. There’s a 100% chance that any webserver or CDN will serve that .js file as text/javascript or application/javascript, and therefore my .js file will be importable into an ES module in browsers. By contrast, if I name my file with .mjs there’s a good chance that a webserver or CDN won’t know what MIME type to use for the file, serving it as text/plain or application/octet-stream or something else unexpected, and then browsers would throw an error when trying to import it. As a package author, I don’t know the environment in which my code will ultimately be executed, so I’m going to choose what’s most likely to work; which is .js.

I could publish with both extensions, of course, to cover both legacy webservers that don’t know what .mjs is and also new ones that do. But then things really get messy for Node: my .js file might have import and export statements in it, but Node thinks it’s a CommonJS file!

This is what I mean when I’m saying that treating .js as CommonJS, when webservers and browsers don’t, is a recipe for incompatibility. It’s not just edge cases of strict mode versus sloppy mode, it’s that the browser and webserver world has _incentives_ to use .js for ESM. And many of them will, because they’re already doing it and they don’t care about Node, and (ironies) the spec allows them to.

One last thought: From browsers’ perspective, ideally Node would’ve created an extension _for CommonJS,_ like .cjs. Then I could publish jquery.js as ESM for browsers (or Node) and jquery.cjs as CommonJS for legacy Node. I’m not actually suggesting we do this, but this is the kind of response I would expect from browser and webserver vendors if they’re ever asked to start treating .js as CommonJS.

GeoffreyBooth on 4 Jul 2018

I think this logic is flawed. Just because there’s no official spec that governs what MIME type webservers should use for .js, it’s not like there’s any doubt what the de facto standard is: text/javascript or application/javascript, which are interchangeable (the former is deprecated by the latter, according to the MIME spec). There’s no chance that webservers will change from this behavior, since switching .js to application/node would be a breaking change (any ESM importing .js in browsers would break) and there’s no benefit to webservers or browsers in using a MIME type that no browsers recognize or are ever likely to recognize.

No one is asking webservers to change the behavior for Script/Module files.

I once again point you to the official MIME registry which is where MIMEs are determined to be safe for use https://www.iana.org/assignments/media-types/media-types.xhtml .

So Node treating .js as application/node goes against this pretty universal de facto standard across probably all real world webservers. And there’s a real consequence, as browsers will load .js files differently than Node will. That, to me, is a spec violation.

Again; there is no spec, this isn't a browsers it is about servers, and we are not requesting you change MIMEs for Script/Module files. You need to use clarification in what is being talked about but I feel like phrases are just being reused even though they don't make sense.

Node has been treating things as non-text/javascript compliant on the server for its entire existence. We want to ensure that we don't treat those files as text/javascript just as much as we want some way to load .js as text/javascript.

The defacto understanding of .js on npm and in Node is not text/javascript.

This is what I mean when I’m saying that treating .js as CommonJS, when webservers and browsers don’t, is a recipe for incompatibility. It’s not just edge cases of strict mode versus sloppy mode, it’s that the browser and webserver world has incentives to use .js for ESM. And many of them will, because they’re already doing it and they don’t care about Node, and (ironies) the spec allows them to.

I completely agree, and it is a large reason we should support things like https://github.com/nodejs/node/pull/18392 , but we cannot change the defacto inside Node without opting out. I don't think anyone here is opposed to allowing .js to be ESM, just that we have a bunch of problems if it is the default. Just add a "mode" or w/e flag to the package.json if you have one and move forward.

One last thought: From browsers’ perspective, ideally Node would’ve created an extension for CommonJS, like .cjs. Then I could publish jquery.js as ESM for browsers (or Node) and jquery.cjs as CommonJS for legacy Node. I’m not actually suggesting we do this, but this is the kind of response I would expect from browser and webserver vendors if they’re ever asked to start treating .js as CommonJS.

I completely agree on this as well, this backwards compatibility problem and .js not being compatible with ESM by default in Node is a problem that we cannot get around easily.

bmeck on 4 Jul 2018

I once again point you to the official MIME registry which is where MIMEs are determined to be safe for use

As I wrote, that registry only defines MIME types, as in, “if you see application/javascript, that’s JavaScript.” It assumes you’re starting with a MIME type, not asking what MIME type to use for a particular file extension. It doesn’t answer the question “what MIME type should I use for .js” and can’t, because there are multiple answers to that question depending on context. In the context of Node in CommonJS mode, the answer is application/node or CommonJS. In the context of webservers serving .js files to browsers, the answer is application/javascript.

In the context of Node in ESM mode, if we want compatibility with the Web we should also choose application/javascript, a.k.a. load .js files as ESM.

The defacto understanding of .js on npm and in Node is not text/javascript. . . . we cannot change the defacto inside Node without opting out.

I’m not suggesting Node change how it loads .js files across the board. That would be a huge breaking change. I’m only discussing within the context of ESM mode. Within a file parsed as ESM, when an import statement specifies a .js file, that file should be loaded as ESM.

There’s also not exactly a de facto understanding of .js meaning CommonJS even within Node. There are thousands of projects that have been written since 2015 where .js files contain import and export statements, and they’re transpiled into CommonJS for execution. That’s the standard for Meteor apps, for example. It’s fairer to say that the extension .js just means “it’s some type of JavaScript” and the module mode is ambiguous from the .js extension alone.

Node has been treating things as non-text/javascript compliant on the server for its entire existence.

Yes, and its entire existence so far has been in CommonJS mode, and CommonJS is a Node-specific thing that doesn’t need to be compliant. But in the brave new standardized world of ESM, Node needs to follow the standards of how the rest of the ecosystem works, or otherwise there’s no point in Node adopting ESM. Node already has a nonstandard module system in CommonJS; it doesn’t need another.

I don’t think anyone here is opposed to allowing .js to be ESM, just that we have a bunch of problems if it is the default.

First, it should only be the default when the code doing the importing is ESM; and second, the fact that that introduces problems is not a reason that we can ignore the standard behavior as we see it in webservers and browsers. We just need to find another solution for importing CommonJS files:

import.meta.require, and we’re okay polluting the import.meta namespace.
import { importCommonJS } from 'module'; importCommonJS(import.meta, './commonjs-file.js'); and we have awkwardness.
We simply don’t allow importing CommonJS files into ESM by default in core Node. If you want to do it, you use a loader that tells Node to treat .js in ESM import statements as CommonJS, or a package.json flag does so, or a CLI flag does so.
This is only half-serious, but we _could_ say that import statements can import CommonJS only when the CommonJS file in question has a .cjs extension, i.e. import './file.cjs'. _That_ would be fully compliant with not just specs but also de facto webserver standards, and we could register .cjs to mean application/node and someday webservers might actually serve it that way (as there’s no other competition for .cjs) and then there’s even a path for browsers to potentially support CommonJS. In practical terms, these .cjs files could be either literal symlinks to .js files or one-liners like require('./file.js'). Obviously this is the least useful solution of all, as it doesn’t help with all the .js files already out there, but it _is_ a way to allow import statements to pull in CommonJS files without any compatibility problems.

GeoffreyBooth on 4 Jul 2018

Browsers ignore extensions entirely. Can we please stop saying that anything related to extensions is “standard behavior” in browsers?

In webservers, putting .js files in script tags is standard behavior. This means Script. The MIME type is imo also unrelated.

ljharb on 4 Jul 2018

👍1

@ljharb Browsers may ignore extensions, but webservers don’t. Webservers use extensions to choose the MIME type that browsers use to decide how to load a file. I explained this above in https://github.com/nodejs/modules/issues/142#issuecomment-402367472. Webservers also serve the files that browsers load via import statements, and that’s where MIME types matter. We’re not discussing <script> tags here.

GeoffreyBooth on 4 Jul 2018

I agree! That’s why discussing browser compat as it relates to extensions is meaningless. We can talk about webserver compat, tho!

ljharb on 5 Jul 2018

We are discussing compatibility with webservers:

https://github.com/nodejs/modules/issues/142#issuecomment-401557098
https://github.com/nodejs/modules/issues/142#issuecomment-401575125
https://github.com/nodejs/modules/issues/142#issuecomment-401825076
https://github.com/nodejs/modules/issues/142#issuecomment-401840289
https://github.com/nodejs/modules/issues/142#issuecomment-401850560
https://github.com/nodejs/modules/issues/142#issuecomment-401871848
https://github.com/nodejs/modules/issues/142#issuecomment-402027006
https://github.com/nodejs/modules/issues/142#issuecomment-402176575
https://github.com/nodejs/modules/issues/142#issuecomment-402367472
https://github.com/nodejs/modules/issues/142#issuecomment-402464404
https://github.com/nodejs/modules/issues/142#issuecomment-402519530

Granted, that’s a lot of discussion to follow. But I think we’re closing in on several points of agreement. Perhaps when this discussion wraps up @bmeck and I can try drafting a summary document.

GeoffreyBooth on 5 Jul 2018

It assumes you’re starting with a MIME type, not asking what MIME type to use for a particular file extension. It doesn’t answer the question “what MIME type should I use for .js” and can’t, because there are multiple answers to that question depending on context. In the context of Node in CommonJS mode, the answer is application/node or CommonJS. In the context of webservers serving .js files to browsers, the answer is application/javascript.

I disagree absolutely on webservers needing/should serve .js with any specific type. If the .js file is written as CJS it still should not be served as text/javascript (WHATWG doesn't want people to use application/javascript). The problem using file extensions is like you point out, there are multiple contexts and differing answers. In Node it is already occupied as application/node and we need to disambiguate files that are intended to be used in Node vs those wishing to have different behavior from Node's ecosystem. As @weswigham points out these servers are configurable, and as I have an example above it is often even configurable on a per file level, some MIME databases also include some heuristics as pointed out above depending on the contents of a file but it is generally unreliable.

We need an out of band mechanism to disambiguate these, there is no single way to state a file is in one context because we have 2 existing and thriving ecosystems in conflict.

I’m not suggesting Node change how it loads .js files across the board. That would be a huge breaking change. I’m only discussing within the context of ESM mode. Within a file parsed as ESM, when an import statement specifies a .js file, that file should be loaded as ESM.

I disagree heavily here. I'm not sure the reasoning on why it should load in a way that changes the understanding of the context that is being used here. As you point out above, Node treats .js as application/node.

There’s also not exactly a de facto understanding of .js meaning CommonJS even within Node. There are thousands of projects that have been written since 2015 where .js files contain import and export statements, and they’re transpiled into CommonJS for execution. That’s the standard for Meteor apps, for example. It’s fairer to say that the extension .js just means “it’s some type of JavaScript” and the module mode is ambiguous from the .js extension alone.

These examples are of preprocessing, all of those files must be compiled to CJS in order to be loaded into Node. I'm not swayed by this because at the time that Node handles these files they are CJS. We cannot handle all possible ways in which people may wish to preprocess files, we are only concerned about the point in which Node handles them.

Yes, and its entire existence so far has been in CommonJS mode, and CommonJS is a Node-specific thing that doesn’t need to be compliant. But in the brave new standardized world of ESM, Node needs to follow the standards of how the rest of the ecosystem works, or otherwise there’s no point in Node adopting ESM. Node already has a nonstandard module system in CommonJS; it doesn’t need another.

"rest of the ecosystem" is problematic. Node and npm already have thriving ecosystem based upon CJS and are heavily used even on the web platform through bundlers. I don't see why that ecosystem isn't being taken into consideration when it also affects what is being deployed to the web platform through bundlers already.

First, it should only be the default when the code doing the importing is ESM;

I disagree heavily here. Since it as you pointed out above .js already has a well understood MIME in Node already. I think the breaking change you are wishing to claim as the default needs more care about the state of the world in both Node and how applications are deployed to the web platform today.

and second, the fact that that introduces problems is not a reason that we can ignore the standard behavior as we see it in webservers and browsers. We just need to find another solution for importing CommonJS files:

As we have been consistently pointing out, there is a convention but no standard involved here. I'm not convinced of any line of reasoning that leads to us needing another solution for importing, but would be ok with it if it was sufficent to preserve the meaning of existing ecosystem files. The ones you have listed out all are problematic to me since they make the meaning of any given file named *.js unclear on if it is application/node or text/javascript. I'm fine letting *.js mean both, but not at the same time.

bmeck on 6 Jul 2018

I disagree heavily here. I’m not sure the reasoning on why it should load in a way that changes the understanding of the context that is being used here. As you point out above, Node treats .js as application/node.

Actually, I said that Node _in a CommonJS context_ treats .js as application/node. We haven’t yet decided on how .js should be treated in an ESM context. Per convention, webservers and browsers have decided that .js in an ESM context should be treated as text/javascript. Node could decide the same.

Let’s take a look for a second at https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.js, the jQuery hosted by Google Hosted Libraries. Google serves this file with a header Content-Type: text/javascript; charset=UTF-8. It starts with this code:

( function( global, factory ) {

    "use strict";

    if ( typeof module === "object" && typeof module.exports === "object" ) {

        // For CommonJS and CommonJS-like environments where a proper `window`
        // is present, execute the factory and get jQuery.
        // For environments that do not have a `window` with a `document`
        // (such as Node.js), expose a factory as module.exports.
        // This accentuates the need for the creation of a real `window`.
        // e.g. var jQuery = require("jquery")(window);
        // See ticket #14549 for more info.
        module.exports = global.document ?
            factory( global, true ) :
            function( w ) {
                if ( !w.document ) {
                    throw new Error( "jQuery requires a window with a document" );
                }
                return factory( w );
            };
    } else {
        factory( global );
    }

Here jQuery is detecting whether it’s in a CommonJS or browser environment, and behaves accordingly. It would be quite a challenge for heuristics to scan this file to determine whether it should be treated as application/node or text/javascript. Really all we know from reading the code is that this file _could_ be loaded as application/node or it _could_ be loaded as text/javascript. That’s the same information we get from the fact that the file extension is .js.

But surely ESM complicates this, right? Even though jQuery manages to produce a single .js file that’s parseable in both CommonJS and browser Script modes, you couldn’t do that _and_ be importable into ESM, could you? Well, we can test it:

<html>
  <head>
    <script type="module">
      import 'https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.js';
      jQuery('body').html('jQuery loaded!');
    </script>
  </head>
  <body></body>
</html>

Yes, this loads jQuery and prints jQuery loaded! in the body. See for yourself. Therefore this jquery.js file is parseable as both strict-mode Script and Module, both CommonJS and ESM. And the jQuery on NPM is identical to this file served by Google. So not every .js file in NPM is intended to be CommonJS—or at least, not _only_ CommonJS.

Now it’s certainly fair to claim that most .js files in NPM modules _could_ be loaded into a CommonJS environment; there are probably relatively few .js files in NPM that execute _only_ in browsers but not in Node, but the number of such files certainly isn’t trivial since Bower shut down and told people to use NPM instead. The NPM modules intended for browser use may even have some hints at CommonJS, like jQuery does, so that bundlers can work with them more easily; but it’s a stretch to say that they’re CommonJS files. Many of them are never intended to be run in a CommonJS environment except as part of bundling; or they’re detecting for a CommonJS-like environment such as that supplied by something like Browserify or RequireJS.

Unfortunately, the presence of a .js extension—even from files loaded from the NPM registry—just isn’t enough to disambiguate files intended for importing into CommonJS versus files intended for importing into a browser environment or ESM (either Node ESM or browser ESM).

We need an out of band mechanism to disambiguate these, there is no single way to state a file is in one context because we have 2 existing and thriving ecosystems in conflict.

Since a single file can be both CommonJS and ESM at once, as we see above with jquery.js, we can’t rely on _anything_ inside a .js file or in its filename/extension to disambiguate it. That’s why relying on .js meaning CommonJS, even in the context of Node and even in the context of files loaded from the NPM registry, is unsafe. It may be correct most of the time, but “most of the time” is all the more dangerous because then edge cases are truly surprising.

Before we dive into solutions, can we at least agree on this? That Node allowing import './file.js' to _by default_ assume that file.js is CommonJS based solely on the .js extension is both an unsafe assumption and an incompatibility with how browsers and typical webservers operate?

GeoffreyBooth on 7 Jul 2018

being correct 90% of the time is better than being wrong 90% of the time, and we all know there are far more cjs modules than anything else on npm.

the goal of the node.js module group should be to support node.js users first. once we verify an idea works for node.js users in the common case we can work into interoperability and whatever else.

devsnek on 7 Jul 2018

👍1

Actually, I said that Node in a CommonJS context treats .js as application/node. We haven’t yet decided on how .js should be treated in an ESM context. Per convention, webservers and browsers have decided that .js in an ESM context should be treated as text/javascript. Node could decide the same.

One of the reasons for my disagreement is a I don't see a differing require() and import context I only see a Node context. Having a single file differ based upon method of consumption is something I was arguing against above.

Really all we know from reading the code is that this file could be loaded as application/node or it could be loaded as text/javascript. That’s the same information we get from the fact that the file extension is .js.

This is not the same information, the UMD wrapper is a well known method to feature detect the environment a Script is being run in and is compatible with multiple environment. The script is doing the side effects after detecting the environment it runs at. This is after it is given a MIME and is being evaluated. We have been arguing about MIMEs here and how sometimes multiple MIMEs can apply to a single source text.

You don't need such a complex example to show that some source texts can run in different modes, a simple console.log(123); is doing the same in being able to run in all sorts of ways.

Unfortunately, the presence of a .js extension—even from files loaded from the NPM registry—just isn’t enough to disambiguate files intended for importing into CommonJS versus files intended for importing into a browser environment or ESM (either Node ESM or browser ESM).

For all of Node's existence it has been application/node. I agree that file extension is not enough, but I firmly believe you have to opt out of the existing behavior, not opt out of new differing behaviors.

Since a single file can be both CommonJS and ESM at once, as we see above with jquery.js, we can’t rely on anything inside a .js file or in its filename/extension to disambiguate it. That’s why relying on .js meaning CommonJS, even in the context of Node and even in the context of files loaded from the NPM registry, is unsafe.

How so, you have only shown that there are source texts that work between different formats, we have not proven that formats themselves are irrelevant. The fact that the source text is designed to work in different formats doesn't really mean it mustn't be one, but instead just shows that it can be any of the supported ones, such as application/node. You have shown that .js can have a source text that works in multiple formats, but I don't follow the requirement that we cannot use a default MIME for the file extension. You even argue that we should use a default MIME of text/javascript.

It may be correct most of the time, but “most of the time” is all the more dangerous because then edge cases are truly surprising.

I'm not sure we can have a 100% foolproof solution without defining the disambiguation mechanism. As you have shown and others, there are a plethora of source texts that can run in multiple formats, and some that even result in different execution. The solution here is to stay unambiguous, and defaulting the MIME to a specific format is likely part of that solution.

Before we dive into solutions, can we at least agree on this? That Node allowing import './file.js' to by default assume that file.js is CommonJS based solely on the .js extension is both an unsafe assumption and an incompatibility with how browsers and typical webservers operate?

I agree that there is a compatibility concern due to the ambiguity. I do not agree that assuming it is CJS is unsafe because that is how Node treats all .js files currently and it isn't having problems either in Node, UMD examples you show above, or bundlers if it stays that way. Node is its own context and still needs disambiguation, webservers could use whatever mechanism but that requires extra effort beyond the dumb servers we have been talking about. We already have these concerns you have when CJS files are served as text/javascript though so I'm not sure how the inverse doesn't apply to .js being non-CJS being incompatible with Node.

bmeck on 7 Jul 2018

I do not agree, because it is not how browsers operate - browsers ignore the file extension, and require an out of band mechanism (the “type” attribute on a script tag) to differentiate. node choosing the file extension as that mechanism is 100% compatible with browsers, and also with webservers that already use file extensions to differentiate file types.

ljharb on 7 Jul 2018

node choosing the file extension as that mechanism is 100% compatible with browsers, and also with webservers that _already_ use file extensions to differentiate file types.

Browsers aren’t really the issue here, webservers are. As discussed above, most (if not all) webservers will choose the same MIME type for both .js and .mjs, causing browsers to load both types of files as ESM; and this isn’t likely to change. This is the incompatibility with Node treating .js and .mjs as meaning different things, since the same is not true of most webservers.

I agree that there is a compatibility concern due to the ambiguity. I do not agree that assuming it is CJS is unsafe because that is how Node treats all .js files currently and it isn’t having problems either in Node, UMD examples you show above, or bundlers if it stays that way.

So we at least agree that there’s a problem here, and the issue is what to do about it, which may be nothing. I’ll try to find time on Monday or Tuesday to condense this discussion into a summary that the group can digest.

GeoffreyBooth on 8 Jul 2018

The concern isn't the MIME type, it's what script tag "type" they'll choose, since that's the mechanism the browsers use to determine how an entry point is parsed.

ljharb on 8 Jul 2018

The concern isn’t the MIME type, it’s what script tag “type” they’ll choose, since that’s the mechanism the browsers use to determine how an entry point is parsed.

The discussion above is around import statements that import .js files. We’re already in ESM mode, in both Node and browsers, to be able to run an import statement at all; the type attribute isn’t relevant by that point. The incompatibility is that an import statement of a .js file will load the file as CommonJS/Script in Node --experimental-modules and as ESM/Module in browsers (as served by a typical webserver).

Until I write up a summary, you can just look at the demo, including reading the code in that repo. That explains the incompatibility pretty succinctly, and it has working code you can run. The webserver in that demo behaves as a typical real-world webserver would.

GeoffreyBooth on 8 Jul 2018

👍2

Yes, of course - but you're assuming no build process. Before anything gets to the browser, the files can easily have been preprocessed to do the right thing.

ljharb on 8 Jul 2018

Let me also respond to some of @bmeck’s points:

One of the reasons for my disagreement is a I don’t see a differing require() and import context I only see a Node context. Having a single file differ based upon method of consumption is something I was arguing against above.

I think I’m finally starting to understand where you’re coming from. I suppose that under the hood, Node’s implementation of loading files probably shares a lot of code in common between import and require, so I can see why as a Node developer you’d see them as nearly equivalent; but from my perspective as a user they’re very different. As a user, I wouldn’t expect require in CommonJS mode to behave equivalently as import in ESM mode. If that _was_ the expectation, then if import './commonjs-file.js' works then so should require('./esm-file.mjs').

Having a single file differ upon method of consumption already happens, at least for initial entry points. Take my demo’s ambiguous.js file. If you have a browser load it via <script src="ambiguous.js"> you get sloppy mode, whereas via <script type="module" src="ambiguous.js"> you get strict mode. So I guess the question is, if the method of consumption is relevant for initial entry points, why shouldn’t it be relevant everywhere?

For all of Node’s existence it has been application/node. I agree that file extension is not enough, but I firmly believe you have to opt out of the existing behavior, not opt out of new differing behaviors.

The user is opting into new behavior by using the import statement. We’ve never had the import statement before, and it’s not equivalent to require; it doesn’t simply inherit require’s existing API. By opting into using the import statement, users are opting into its new syntax and new behaviors.

I’m not sure we can have a 100% foolproof solution without defining the disambiguation mechanism. As you have shown and others, there are a plethora of source texts that can run in multiple formats, and some that even result in different execution. The solution here is to stay unambiguous, and defaulting the MIME to a specific format is likely part of that solution.

Truly staying unambiguous would mean that you would never need to disambiguate.

Unambiguous on the author side would be .mjs for ESM and .cjs for CommonJS. I just don’t think we can really safely assume anything about the author’s intent from a .js extension. By definition, .js is ambiguous.

Unambiguous on the consumer side would be having import only ever import ESM files into ESM mode, like what browsers do. If one wants CommonJS-into-ESM import interoperability, one could use a separate API to achieve that, such as import.meta.require. It’s hard to get more unambiguous than completely separate APIs for importing CommonJS versus importing ESM; I think that’s why @MylesBorins was assuming that import.meta.require would be uncontroversial, since you could always have that _in addition_ to a disambiguating import statement. Having separate APIs seems to be the plan for CommonJS, where the leading proposal seems to be using import() for importing ESM into CommonJS, rather than having require disambiguate CommonJS from ESM by looking for an .mjs extension.

GeoffreyBooth on 8 Jul 2018

👍1

I think I’m finally starting to understand where you’re coming from. I suppose that under the hood, Node’s implementation of loading files probably shares a lot of code in common between import and require, so I can see why as a Node developer you’d see them as nearly equivalent; but from my perspective as a user they’re very different. As a user, I wouldn’t expect require in CommonJS mode to behave equivalently as import in ESM mode. If that was the expectation, then if import './commonjs-file.js' works then so should require('./esm-file.mjs').

They actually share very little code, this is coming from someone who has been authoring both ESM and CJS in my Node applications. Having a file of unknown format is problematic, for tooling, dealing with issues around teaching, defensive programming, etc.

My expectation as an author is that my file doesn't get interpreted in different ways than what I intended it to be interpreted as. That applies to execution and linking, both forwards compatibility and backwards compatibility.

Having a single file differ upon method of consumption already happens, at least for initial entry points. Take my demo’s ambiguous.js file. If you have a browser load it via .

Indeed this is part of why having ambiguity is problematic. I can create a file:

// test.wasm
console.log(123);

serve it with application/wasm and it still runs in a <script> tag as a Script. Your point about serving Script as text/javascript doesn't apply to anything because no means of loading that check text/javascript load as a Script. I was trying to point out if there is a claim of ambiguity around text/javascript between files and loading as Script vs Module, the same ambiguity exists for any MIME including things like WASM. You can serve it so that it loads in type=module using some format, but it always collides with the Script goal of JS due to <script>. I don't see how this claim of ambiguity only affecting .js files is true.

Because otherwise we have a major incompatibility with the Web.

I still don't see the incompatibility issue at heart here. We can still have things treated as text/javascript if you opt-in. In addition, the claim of incompatibility is weak in my eyes because it relies on <script> which as I said earlier applies to all files not just .js files that are served with text/javascript.

There certainly is a difference in default behavior if we choose to continue treating .js as application/node, but I don't see any incompatibility that prevents people from writing code that works on both platforms.

If I as a package author want to publish a JavaScript Module library for wide use on the Web, I want to publish it as a .js file because that’s much more compatible across the webservers of the world than .mjs is.

This seems a fine goal, but isn't necessarily related to the default behavior of the web nor node. This could be an opt-in thing and I'm unsure why it being opt-in is problematic when we are already using non-web compatible features like package.json in our ecosystem.

That’s why it wasn’t a mistake for WHATWG to reject the new MIME type: the benefits of author-specified unambiguity are outweighed by the cost of all the webservers of the world needing updates to serve a new file extension with a new MIME type.

text/javascript has never had significant meaning on web browsers. text/javascript is only used in determining if something is a Module never a Script on web browsers. The MIME remains unambiguous on web browsers, but it seems like you think it means Script as well for browsers, and if so can you clarify how browsers are using it related to the Script goal.

A similar cost/benefit analysis applies to Node: Node gets some benefits, sure, from author-specified unambiguity; but the cost is incompatibility with .js ESM files, which the Web allows and encourages, and of which there will soon be many as ESM support in browsers becomes widespread.

I think as long as opt-in to treat .js as text/javascript is easy or even automated in some way it isn't costly at all. I, however, am seriously concerned with the idea of not being able to statically determine what format a file is in. Easier and less bug inducing for the author to clarify intent than us to make things muddy. As long as the mechanism to enable your use case is simple and efficient I don't see how this difference in defaults should be blocking. A difference in defaults makes sense, hence why Node was given a standards track MIME instead of a vendored MIME. Node has a significant existing ecosystem, and opting into different behavior could be as easy as adding a "mode" flag to your package.json. In addition, things that disambiguate in a user configurable fashion allow people to put JSX/Flow/etc. in their .js files while remaining unambiguous (as long as JSX/Flow/etc. make a MIME for their format).

Basically, Node doesn’t need author-defined file-level unambiguity.

I believe the loss of static guarantees about how files are intended to be run is enough to make it a need even if you disagree.

Consumer-defined disambiguation can work, though you may not prefer its syntax.

I don't think any of my comments so far have been about syntax needing to be a specific way. They are rooted in ambiguity problems.

If I had to choose between conflicting goals of allowing authors to enforce the parse goal of their file, versus more compatibility with the Web, I would choose the latter. Authors can always informally specify the parse goal of their file, via filenames like foo.esm.js, to signal to consumers how a file should be consumed. I don’t think the enforcement is all that valuable or even desirable.

You can have both using an opt in mechanism. I don't understand this comment.

bmeck on 9 Jul 2018

I think as long as opt-in to treat .js as text/javascript is easy or even automated in some way it isn’t costly at all.

If there’s a way to have Node treat .js files as ESM, yes, the incompatibility goes away. Then the question becomes how that should be implemented, and whether or when it should be the default. This feels like a great stopping point to shift that discussion into a new thread.

GeoffreyBooth on 9 Jul 2018

If there's a way for node to treat .js files as ESM, shouldn't there be a way to treat .js files as WASM, and .wasm files as CJS, and .anything files as "anything"? (i realize this may come off as a sarcastic question, but it's a genuine one)

ljharb on 9 Jul 2018

@ljharb I would assume so, yes. In particular I'm interested in handling things like Flow/JSX/etc. that also live in .js.

bmeck on 9 Jul 2018

👍2

The fact that their spec uses .js in its examples is proof enough that they don’t encourage file extension disambiguation.

The HTML Standard uses both .mjs and .js in its examples for module scripts, as well as URLs without a extension or ending with .cgi. This matches reality where on the web, file extensions don’t matter at all to user agents; HTTP headers do.

But how does one decide which HTTP headers to send out? In practice, it happens based on the file extension as opposed to on a file-by-file basis. In general, you’re gonna have an easier time during development but also when configuring your server by using .mjs for modules and .js for scripts and sticking to it consistently.

mathiasbynens on 9 Jul 2018

❤2 👍2

This topic seems to have cooled and is being addressed on a per phase basis.

bmeck on 30 Dec 2018

Modules: disambiguate "web compatibility"

Most helpful comment

All 113 comments

Related issues