Modules: Simplifying package.json of backward-compatible packages (type: "module")

[sheerun]

As asked by @GeoffreyBooth in #323 I'm opening new issue for this discussion

The "type": "module" solution for publishing MJS modules had an issue. Specifically was not possible to publish package that cold be simultaneously consumed in following ways (without publishing separate package like lodash-es):

1. const lodash = require('lodash') in old node (for projects without upgraded node)
2. const lodash = require('lodash') in new node (for projects without upgraded code)
3. import lodash from 'lodash' in new node (for projects with upgraded node and code)

According to interesting comment by @jkrems the current experimental solution for this is to run node with --experimental-conditional-exports and publish lodash with following package.json

{
  "main": "./old-node.cjs",
  "exports": {
    ".": {
      "require": "./old-node.cjs",
      "default": "./new-node.js"
    }
  },
  "type": "module"
}

Needless to say this seems quite complicated package.json what should probably be the default for all migrated packages.

Additionally it requires lodash to rename all its current CJS-compatible files to .cjs extension, while the current standard practice is to put all modern code to one directory (e.g. src), and all compiled code to other (e.g. lib). Additionally "type": "module"

I'm opening this issue to suggest that this issue needs to be solved before these features go stable, discuss this issue, and suggest one solution myself. Ideal solution would:

The solution I suggest is to:

1. Leave main to mean CJS compatible entrypoint without ability to change its meaning with "type": "module". This ensures 1. and 2. can be always handled properly.
2. When consumer uses import instead of require, make node use already implemented "conditional-exports" logic and look at exports field of package.json + ignore main field

Without "type": "module" there is no need for conditional conditional-exports because they are currently used to override back what it changes. Even worse, using conditional-expressions in any form for CJS modules makes these changes not backward compatible, because old nodes would not know how to interpret this.

The final package.json would be as follows (thanks to already implemented exports sugar):

{
  "main": "lib/index.js",
  "exports": "src/index.js"
}

which looks like much better alternative. Additionally if lodash would like to support importing subpaths like import first from 'lodash/first', the package.json is equally simple:

{
  "main": "lib/index.js",
  "exports": "src/"
}

[ljharb]

Packages don’t just have one entry point; they have infinite - as such, a map of exports is always needed.

Using only the directory doesn’t allow for selective exposing/hiding of files within the same directory.

[sheerun]

I'm not asking to do anything (or remove) support for exports field in package.json as designed currently, but rather to not make "type": "module" stable and instead deprecate it.

[guybedford]

@sheerun you do not need to use type module if you don't want to.

An alternative configuration to do the same thing would be the following:

{
  "main": "./cjs-main.js",
  "exports": {
    "require": "./cjs-main.js",
    "default": "./esm-main.mjs"
  }
}

Note that CommonJS supports "exports" so the whole point of these schemes exactly is to be explicit about the fact that there is a bifurcation happening here.

[sheerun]

@guybedford Why can't I write this as follows:

{
  "main": "./cjs-main.js",
  "exports": {
    "default": "./esm-main.mjs"
  }
}

Which can be sugared to follows?

{
  "main": "./cjs-main.js",
  "exports": "./esm-main.mjs"
}

[GeoffreyBooth]

Additionally it requires lodash to rename all its current CJS-compatible files to .cjs extension, while the current standard practice is to put all modern code to one directory (e.g. src), and all compiled code to other (e.g. lib). Additionally "type": "module"

This isn’t correct. A package can have multiple package.json files in different folders; each one starts a new package scope. So for your example, you could have:

/package.json:      "type": "module"
/dist/package.json: "type": "commonjs"

And then all .js files in or under /dist would be treated as CommonJS, while all .js files elsewhere would be treated as ES modules.

The .cjs and .mjs extensions are only necessary if you want to intermix CommonJS and ES module files within the same folder.

[GeoffreyBooth]

Why can't I write this as follows:

Because "exports" is not exclusive to ES modules. If your package gets used via require, what’s defined in "exports" will be used. "exports" overrides "main".

[sheerun]

@GeoffreyBooth But I don't use require in exports, so it should fallback to main for this kind of import. At least it would be a good default behavior to avoid duplication in package.json

[GeoffreyBooth]

@GeoffreyBooth But I don't use require in exports, so it should fallback to main for this kind of import

That’s not how "exports" is designed. "exports" applies equivalently to CommonJS and ES modules. If a version of Node that supports "exports" is being used, what’s in "exports" will take precedence over "main" for both CommonJS and ES modules. The only time anything defined within "exports" differs between the two modes is via --experimental-conditional-exports.

And before you ask why can’t this _not_ be the case, why does "exports" need to apply to CommonJS at all, the answer is because the group decided to preserve CommonJS as an equal to ES modules within Node. So whatever features get added to ES modules get supported in CommonJS as well, whenever possible.

The next version of Node includes a new section in the docs with recommended approaches for publishing what I call dual packages, packages meant to be used by both CommonJS and ES module consumers: https://github.com/nodejs/node/blob/master/doc/api/esm.md#dual-commonjses-module-packages

[sheerun]

Could you update this guide to tell how to create Dual Package without using .cjs or .mjs extensions and still supporting code that uses require('lodash') and require('lodash/first') on older node (and node that supports exports)? Even better publish some example "upgraded" package as an example?

[ljharb]

You can’t have a dual package without two extensions, or, without using multiple package.jsons.

(Also as soon as node is released with unflagged modules, there will be many examples to point to)

[sheerun]

You can’t have a dual package without two extensions, or, without using multiple package.jsons.

I hope that unflagged modules will allow to publish single package.json and just tell that some subfolder (e.g. src) contains es6 modules, without using .cjs or .mjs extensions anywhere.

[guybedford]

There's nothing wrong with the approach of putting a package.json with { "type": "module"} in a subdirectory to do the above.

An important property of the system though is that it is possible to determine the module format of a file in this way without having to rely on assuming how the consumer got to the file. This is an important property for a module registry as the registry is a file-pathed namespace.

[sheerun]

So package.json:

{
  "main": "index.js",
  "exports": {
    "require": "index.js",
    "default": "src/index.js"
  }
}

and src/package.json:

{
  "type": "module"
}

?

[guybedford]

Yes exactly.

[sheerun]

Could you just show how it should look like if both packages need to support subpath imports? i.e. what should be package.json and dist/package.json if all of following are possible:

• const lodash = require('lodash') // in old node
• const first = require('lodash/first') // in old node
• const lodash = require('lodash') // in new node
• const first = require('lodash/first') // in new node
• import lodash from 'lodash'
• import first from 'lodash/first'

[ljharb]

For “first”, two of the many possibilities: you could have first.js and an exports entry that pointed to that for require and to an alternative for default; or you could have a first dir with a package.json that only accounted for the main.

[GeoffreyBooth]

Could you update this guide to tell how to create Dual Package without using .cjs or .mjs extensions and still supporting code that uses require('lodash') and require('lodash/first') on older node (and node that supports exports)? Even better publish some example "upgraded" package as an example?

A good tutorial article online is https://2ality.com/2019/10/hybrid-npm-packages.html, and as @ljharb mentions there will surely be more to come in the near future. We don’t plan for the docs to show examples for every use case, as we’re trying to keep them as concise as possible. The section on dual packages is already quite long.

I can try to answer your question here though. First, I wouldn’t recommend publishing a dual package _at all_ until either --experimental-conditional-exports or its replacement is unflagged (ETA January). Everything related to dual packages is very much subject to change until then.

Second, if your package already has a public API like lodash/first and you want to maintain compatibility with old Node, then you can’t put all your CommonJS files in a subfolder like dist/. In today’s Node to make lodash/first work you need to have either a first.js or a first/index.js from the top level of your package. So that narrows down your options, to something like this:

• package.json
• index.js
• first.js, other similar files
• es/package.json
• es/index.js
• es/first.js, other similar files

The root package.json would contain:

{
  "name": "lodash",
  "type": "commonjs", // Not necessary, but encouraged
  "main": "./index.js",
  "exports": {
    ".": {
      "require": "./index.js",
      "default": "./es/index.js"
    },
    "./first": {
      "require": "./first.js",
      "default": "./es/first.js"
    },
    // and so on for every public export

And es/package.json would be, in its entirety:

{
  "type": "module"
}

I hope that unflagged modules will allow to publish single package.json and just tell that some subfolder (e.g. src) contains es6 modules, without using .cjs or .mjs extensions anywhere.

Unfortunately it’s just too complicated to have one package.json define different "type" values for various folders. What would happen if other package.jsons within that tree defined a different "type" for the same folder? We’re also trying to keep package.json as simple as we can. Following the pattern I’ve just described, however, you can avoid using .cjs or .mjs extensions.

[sheerun]

Would following be OK?

package.json:

{
  "main": "index.js",
  "exports": {
    "require": "./",
    "default": "src/"
  }
}

src/package.json:

{
  "type": "module"
}

with following files:

package.json
index.js
first.js
src/package.json
src/index.js
src/first.js

[GeoffreyBooth]

Would following be OK?

In your version, ES module consumers would need to type import first from 'lodash/first.js'. If you want just 'lodash/first', you need to explicitly define the ./first export. See my example.

[sheerun]

In your version, ES module consumers would need to type import first from 'lodash/first.js'.

Unless --es-module-specifier-resolution=node will also be a default? It would simplify greatly package.json for this use case.

[GeoffreyBooth]

Unless --es-module-specifier-resolution=node will also be a default? It would simplify greatly package.json for this use case.

It would, but there is still considerable opposition to changing that default within this group. Many members of the group want to encourage public packages to have import references that include extensions, e.g. import './file.js' rather than './file', because the former is much more likely to be usable unmodified in browser environments and we’re trying to encourage cross-compatible code. That’s a higher priority for most people than keeping package.json files as short as possible.

Also few packages have as many public exports as lodash. Most packages seem to have only one, the default export, so cases like lodash are already exceptional.

[sheerun]

Then it means import first from 'lodash/first.js' is better for these group members?

[devsnek]

because the former is much more likely to be usable unmodified in browser environments and we’re trying to encourage cross-compatible code.

but you're also encouraging using package.json mappings, which browsers definitely don't support. I don't understand this dichotomy.

[GeoffreyBooth]

Then it means import first from 'lodash/first.js' is better for these group members?

Not necessarily, because that is in consumer code, outside of the package itself. Think of it this way: to use a package in a browser context, you need to pull in an entry point somehow, e.g.:

import _ from 'https://somewhere.com/lodash/es/index.js';
// or
import first from 'https://somewhere.com/lodash/es/first.js';

The URL you use in a browser is separate from anything defined in package.json.

But then _within_ that index.js or first.js, it’s more compatible to have relative references that browsers can resolve. So for example within index.js, there might be:

export * as first from './first.js';

This will work in browsers, whereas export * as first from './first' will not (unless you have special mappings set up by your webserver).

There’s also something called import maps coming to browsers (in Chrome already) that provides a browser equivalent to "exports"’ path mappings.

[MylesBorins]

Import maps are already shipping behind a flag in chrome and will be
unflagged in the not too distant future.

Anything defined in an export map will be able to be used to statically
generate an import map 🎉

On Sat, Nov 16, 2019, 1:24 PM Geoffrey Booth [e-mail]
wrote:

Then it means import first from 'lodash/first.js' is better for these
group members?

Not necessarily, because that is in consumer code, outside of the package
itself. Think of it this way: to use a package in a browser context, you
need to pull in an entry point somehow, e.g.:

import _ from 'https://somewhere.com/lodash/es/index.js';
// or
import first from 'https://somewhere.com/lodash/es/first.js';

The URL you use in a browser is separate from anything defined in
package.json.

But then within that index.js or first.js, it’s more compatible to have
relative references that browsers can resolve. So for example within
index.js, there might be:

export * as first from './first.js';

This will work in browsers, whereas export * as first from './first' will
not (unless you have special mappings set up by your webserver).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/432?email_source=notifications&email_token=AADZYV725AUDILG4XGILBADQUBQHPA5CNFSM4JOE2K62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEH24II#issuecomment-554675745,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADZYV5PQLFB545ZZVRAPMTQUBQHPANCNFSM4JOE2K6Q
.

[devsnek]

@MylesBorins I'm just trying to figure out what our goals here are. Do we want modules to be usable unmodified in browsers or are we okay with tooling being needed?

[sheerun]

I fail to see where is issue making --es-module-specifier-resolution=node a default given that import maps solve this issue in browsers and they can be statically generated.

Using explicit extensions in consumer code seems like antipattern because such code won't survive or allow refactors in packages themselves (for example: changing .js extension to .mjs or vice versa, or moving utils.js to utils/index.js if it becomes complex enough).

My experience tells me that using node modules in browser environment without tooling can never be performant and is usable only in development, I don't believe this can be solved with modules.

[MylesBorins]

Can we move this to another thread this is off topic.

But tldr there is a huge difference between tooling that can generate meta
data at install time vs transpilation

On Sat, Nov 16, 2019, 2:09 PM Gus Caplan [e-mail] wrote:

@MylesBorins https://github.com/MylesBorins I'm just trying to figure
out what our goals here are. Do we want modules to be usable unmodified in
browsers or are we okay with tooling being needed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/432?email_source=notifications&email_token=AADZYV2DXE2YEQGOFHQUPLTQUBVSHA5CNFSM4JOE2K62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEH3VFA#issuecomment-554678932,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADZYV6QEISHNHAYBG6LP2DQUBVSHANCNFSM4JOE2K6Q
.

[MylesBorins]

Import maps cannot generate file extensions they can allow you to convert a
specifier into a URL.

If you wanted to write code in a repo with 0 file extensions for any import
you would need an entry for every file, which is fairly verbose.

If we turned on the resolution algorithm by default we would lose the
static nature of the current implementation. We would have no way to
determine the full path without multiple pings to the filesystem.

This is actually a non trivial performance issue with nodes cjs algorithm

On Sat, Nov 16, 2019, 2:11 PM Myles Borins [e-mail] wrote:

Can we move this to another thread this is off topic.

But tldr there is a huge difference between tooling that can generate meta
data at install time vs transpilation

On Sat, Nov 16, 2019, 2:09 PM Gus Caplan [e-mail] wrote:

@MylesBorins https://github.com/MylesBorins I'm just trying to figure
out what our goals here are. Do we want modules to be usable unmodified in
browsers or are we okay with tooling being needed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/432?email_source=notifications&email_token=AADZYV2DXE2YEQGOFHQUPLTQUBVSHA5CNFSM4JOE2K62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEH3VFA#issuecomment-554678932,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADZYV6QEISHNHAYBG6LP2DQUBVSHANCNFSM4JOE2K6Q
.

[evanplaice]

Isn't the root issue with Node's resolution algoritm? That -- what is cheap and easy ona local FS -- becomes a perf deal breaker in browsers?

[MylesBorins]

It isn't necessarily cheap on the file system either when you scale up to
millions of entry points

On Sat, Nov 16, 2019, 3:39 PM Evan Plaice [e-mail] wrote:

Isn't the root issue with Node's resolution algoritm? That -- what is
cheap and easy ona local FS -- becomes a perf deal breaker in browsers?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/432?email_source=notifications&email_token=AADZYV5XOEONMY3Y7ZLJ3FLQUCADZA5CNFSM4JOE2K62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEH5EHA#issuecomment-554684956,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADZYV2XGP7VKEHUA7GLBITQUCADZANCNFSM4JOE2K6Q
.

[GeoffreyBooth]

I fail to see where is issue making --es-module-specifier-resolution=node a default given that import maps solve this issue in browsers and they can be statically generated.

Import maps are a more-or-less direct equivalent to "exports", in that they allow a mapping of a public API. They’re a poor tool for bringing automatic extension resolution to the Web. As the import maps README puts it:

Although this example shows how it is _possible_ to allow extension-less imports with import maps, it's not necessarily _desirable_. Doing so bloats the import map, and makes the package's interface less simple—both for humans and for tooling.

---

My experience tells me that using node modules in browser environment without tooling can never be performant and is usable only in development, I don't believe this can be solved with modules.

This shouldn’t be the case for too much longer, if it isn’t already a thing of the past. Technologies like HTTP/2 are designed to make the need to bundle JavaScript no longer necessary from a network performance perspective; the idea behind encouraging public packages to be browser-compatible by default is to lessen (or eliminate) the need for bundling for compatibility reasons as well.

Using explicit extensions in consumer code seems like antipattern because such code won't survive or allow refactors in packages themselves (for example: changing .js extension to .mjs or vice versa, or moving utils.js to utils/index.js if it becomes complex enough).

That is the benefit of filenames without extensions, yes. The question is whether that benefit is more important than pushing public package authors toward a future where universal JavaScript is the default, rather than something achieved only through tooling. That’s a subjective call that different people will feel differently about.

And it’s not quite a zero-sum game. A tool like a Babel plugin could certainly rewrite specifiers such as './file' into './file.js' or './file/index.js' as part of a build process. So then an author would still get the refactorability of extensionless specifiers, while Node and the browser community would get the benefits of explicit extensions in published source files. Once native ESM becomes more popular in Node after it ships, I expect to see lots of tools to help with things like this.

This has been a good discussion and a good review of the implementation as it is currently designed, but aside from answering any other questions you may have, is there any specific feedback you’d like to see addressed? Obviously we can’t make everyone happy with every decision we’ve made, but I hope we’ve at least explained them such that you understand why things came out the way they did. I also hope that those design decisions can be ones that you at least can work with, even if they’re not quite what you might have chosen had it been entirely up to you.

[sheerun]

Technologies like HTTP/2 are designed to make the need to bundle JavaScript no longer necessary from a network performance perspective

Performant web apps require much more than simple response multiplexing. They involve tree shaking, minifying, ordering resources in proper order, deduplicating imports, adding polyfills for legacy or mobile browsers, pre-rendering, small image inlining, among others. Processing and bundling on production needs to happen even if it happens on-the-fly.

Although this example shows how it is possible to allow extension-less imports with import maps, it's not necessarily desirable. Doing so bloats the import map, and makes the package's interface less simple—both for humans and for tooling.

If processing needs to happen anyway for production application for the reasons above, either statically or in cached-on-the-fly way, all import paths can be inlined, so even if in code there is import fp from "lodash/fp" the browser can receive import fp from "lodash/fp.js". Import maps are good for avoiding this extra processing, but only in development. I can't see any way performant production app would just import unmodified .js, .mjs, or .cjs files.

A tool like a Babel plugin could certainly rewrite specifiers such as './file' into './file.js' or './file/index.js' as part of a build process. So then an author would still get the refactorability of extensionless specifiers, while Node and the browser community would get the benefits of explicit extensions in published source files.

With this approach ultimately the only reason to build project for node would be to apply node resolution logic which is very weird. And yes, Babel can rewrite ./file to ./file.js for browser so slow import maps are not necessary. But for development building is not needed thanks to them.

This has been a good discussion and a good review of the implementation as it is currently designed, but aside from answering any other questions you may have, is there any specific feedback you’d like to see addressed?

I'd just like to express my concern that unflagging new modules without making --es-module-specifier-resolution=node a default will slowdown or prevent upgrade of existing node packages and applications for following reasons:

1. Need to change extensions of files in repository to either .mjs or .cjs
2. Need to resolve node imports to their exact extensions in code
3. Manually adding exports entries to package.json
4. Keeping track which dependencies are es6 and which are cjs imports
5. Doing everything above gradually because upgrading whole codebase at once is very hard

If performance is concern paths can be rewritten either when bundling (on-the-fly or not) for browser, installing packages with npm/yarn or publishing packages to registry.

[sheerun]

I can suggest one more alternative that seems good for both sides: --es-module-specifier-resolution=js which is would be simplified resolution algorithm that just adds .js extension if no extension is present in import, and given imported name is not in import maps.

So let's say there's following package.json:

{
  "main": "index.js",
  "exports": {
    "require": "./",
    "default": "./src/"
  }
}

• require('lodash') in old node loads lodash/index.js with old rules
• require('lodash/first') in old node loads lodash/first.js with old rules
• require('lodash') in new node loads lodash/index.js because of require in exports
• require('lodash/first') in new node loads lodash/first.js for the same reason
• import lodash from 'lodash' loads lodash/src/index.js
• import first from 'lodash/first' loads lodash/src/first.js

This way there's both no node resolution algorithm and it's possible to not use .js extension.

[sheerun]

It seems that most of this discussion is rendered irrelevant because { "type": "module" }, import, export has been made stable and --es-module-specifier-resolution=node has not been made default. I'll open another issue that focuses just on assuming default extension when importing.

[ljharb]

@sheerun the modules implementation is experimental; there remains the possibility that things like the default specifier resolution could change.