Modules: Phase 2-3: package.json configuration

[GeoffreyBooth]

I did some research into the package.json "module" field. I posted my findings here, including which popular NPM packages are already using it.

My impetus for looking into this was the discussion of potentially using the "module" field as our way to determine the ESM entry point for packages. Basically, if we choose to use "module", there are 941 packages published publicly on the NPM registry that are already using the field. Only 75 of those (8%) point to filenames with an .mjs extension, and could potentially be used with the new modules implementation as it stands today. If .js extensions become allowed in "module", the share rises to 91%. Either way, I can’t tell from package.json alone that these packages are compatible or not with our implementation; even if the entry point is index.mjs, there might very well be incompatible things inside the module such as require statements or import statements of filenames without extensions.

Using "module" has both pros and cons. In its favor, Node’s adoption of "module" builds on some fairly substantial usage already in the community. If Node’s final ESM implementation is compatible or mostly compatible with the packages already using "module", that will presumably help speed adoption of ESM across the Node ecosystem. On the flip side, if Node’s ESM implementation is incompatible with many of the packages already using "module", there could be pain for users as they try to import some of these packages (including some very popular ones like sinon and redux and vue) and Node throws errors. Presumably the most popular packages are very actively maintained and will update their "module" builds before too many users try to use them in native ESM mode, but there’s always the risk that the transition period will be messy.

As an alternative, we could come up with a new package.json field, like mainModule, that isn’t currently used (or not commonly used). But I think a better solution might be to create a new package.json field called nodejs that takes a configuration object:

"nodejs": {
  "main": "./dist-commonjs/index.js",
  "module": "./dist-esm/index.mjs"
}

This namespacing frees us up from worrying about collisions with package.json fields already in use by the community, both now and in the future. It also gives us a place for other potential configuration options, like the mimes proposal, and follows a pattern used by other popular Node modules such as Babel and ESLint and Prettier. Node would control the specification of fields in this block, and tools like Webpack and Rollup would presumably make sure that they follow Node’s rules for things like ESM support before outputting to nodejs.module. Adoption will be slightly slower than if we go with straight "module", but at least we won’t have early-but-incompatible adoption. Thoughts?

cc @guybedford @jkrems @SMotaal

[giltayar]

There is no chance that we are compatible with all those modules. For example, they use import with named imports to import cjs modules, which is currently not allowed by NodeJS, and may not be allowed in the end.

Of course, this may change in the future, but until it does, we should assume incompatibility.

[giltayar]

In phase 2, from what I recall, NodeJS can explicitly know whether you are importing a CJS or an ESM, because if you are using import then you're importing ESM, and if you're using createRequireFunction then you're importing CJS.

And if that is so, why would we need two fields in the package.json? We can stick with the given main field.

And if the answer to that is "for future use", then I suggest we wait for that future (which may never come), and _then_ decide to add this new field.

[robpalme]

@giltayar The type of module the consumer is asking for is definitive. But the target package may want to also support CJS consumers. Effectively a package can have multiple entrypoints and these get instantiated separately, i.e. independent identities. This has been called "dual-mode" packages.

@geoffreybooth what is your proposed behaviour if the "nodejs" field is missing in the target package of an ESM import? I hope we can avoid a cascade.

[ljharb]

@giltayar there is definitely not consensus on that for longer term; i feel very strongly that only the author should ever get to decide the parse goal of a file, and never the user’s choice of consuming it.

The “module” field is tainted because it has nonzero community usage; I’m quite confident we can’t use it. Making a “node” field is a decent idea, and we could bikeshed the name separately. I’m not sure we’d want to put “main” there since it already lives at the top level.

[devsnek]

fields for source types doesn't scale at all. what about wasm and html and etc etc etc etc

[giltayar]

@robpalme. Oops, my bad. Point taken (about dual-mode). And this coming from somebody who next week is giving a talk about ES modules in NodeConf, where I _explicitly_ talk about dual-mode libraries.

Ouch. :-)

[giltayar]

@ljharb - I wasn't saying that we should have author-decided _or_ consumer-decided parse goals (I usually am for author-decided on weekdays, and consumer-decided on weekends :-) ).

All I was saying is that given the incremental nature of our current way of thinking, we shouldn't put the cart before the horse, and that _currently_, we are in a consumer-decided step of our iterative process (the way we parse the js file is decided by the consumer). If I am wrong about that, I'd love to know.

But this is now beside the point, as @robpalme pointed out, given that another possible need for these fields is dual-mode libraries.

[devsnek]

currently, we are in a consumer-decided step of our iterative process (the way we parse the js file is decided by the consumer). If I am wrong about that, I'd love to know.

@giltayar right now we only allow esm in mjs files, so its currently entirely author decided.

[GeoffreyBooth]

@GeoffreyBooth what is your proposed behaviour if the “nodejs” field is missing in the target package of an ESM import? I hope we can avoid a cascade

So currently Node only cares about the "main" field, as far as I’m aware. I was assuming that for whatever the next field or fields we add, we would add them inside this new "nodejs" top-level field and keep things namespaced. If "nodejs" were missing, Node would fall back to its current CommonJS-only behavior. So if you want to signal that your package has an ESM entry point, you would need to define nodejs.module (or whatever it ends up being). Whether we provide _other_ ways to define the ESM entry point, such as an implicit index.mjs file or something, was beyond the scope of what I was intending to propose here.

fields for source types doesn’t scale at all. what about wasm and html and etc etc etc etc

I’m not sure what “fields for source types” means. One thing I forgot to mention is that I hadn’t given much/any thought to the _contents_ of the "nodejs" field. I’m not proposing specifically that it contain "module" and "main", those were meant as an example. I suppose that if a package could theoretically have four or five entry points (after we support WASM and HTML and binary AST and so on) then we should design the field accordingly:

"nodejs": {
  "entrypoint": [
    {"ast": "..."},
    {"wasm": "..."},
    {"esm": "..."},
    {"commonjs": "..."}
  ]
}

Or whatever. Again, just an example. I figure that this is something we could work out in the implementation. We could also add a nodejs.version field to _really_ be future-proof if we want.

[devsnek]

this just seems like a really really poor design path. why do you have 30 entrypoints? why can't they all be resolved via "main": "./src/index"?

this also seems to imply that there's some form of being able to request a module of a certain format... when would the ast or wasm options ever be used there? how would they be used? nothing here makes sense.

this seems to be conflating out-of-band methods of specifying module formats with module resolution, which are two separate things.

[jkrems]

@GeoffreyBooth Is the user story here "I have shipped a rust library before importing WASM directly was a thing. I want to expose the compiled WASM directly for users of the latest node and an ESM wrapper that loads and compiles it manually for older node versions"?

[GeoffreyBooth]

@jkrems That’s one, I suppose. My user story is #151, that I need to use ESM in .js files for compatibility with CoffeeScript; and the same likely applies to most transpiled languages.

Whatever the particular design is, using JSON to define the entry points is far more specific and scalable than "main": "./src/index". The package author could decide the order of preference between various types of entry points, for example. Also all the metadata for the package would be in package.json, which makes generating the package name maps for browser compatibility straightforward.

[devsnek]

@GeoffreyBooth the method for saying what type of file something is has to be separate from resolution anyway, because both deep imports and tooling need to be able to tell what each individual file is.

[GeoffreyBooth]

the method for saying what type of file something is has to be separate from resolution anyway, because both deep imports and tooling need to be able to tell what each individual file is.

That’s fine. You could define both in the package.json "nodejs" object. That object could even tell Node how to handle deep imports. It’s a JSON object, the possibilities are limitless.

Again, the point of this thread was just to propose a place to put metadata/configuration data that describes a package. I don’t have a proposal for a _design_ of the "nodejs" object, because that would entail figuring out answers for all the cases of imports and entry points and so on, and that will evolve as our implementation matures; but I’m sure we can figure out those details in the implementation.

[SMotaal]

Okay, since I made a pact not to emoji (unless intent is clear), and do so aiming to avoid the pitfalls of miscommunication as we work together to make head towards our mutual responsibility.

@GeoffreyBooth provides one detailed path to solve particular problems from the set of problems that need solving. Irrespective of favouring his idea or not, his efforts from a requirements gathering perspective for problem solving serves as one of several alternatives that need to be contrasted against one another to yield 1 of the following outcomes:

1. Alternative x is the most ideal and is a complete solution to the problem.
2. Alternative x is somewhat better than the rest, but lacks aspects which can borrow from y or z.
3. Alternative x is equally as bad as y and z, none of those alternatives solve out fundamental problem.

In the first two, you have a solution to the problem. You proceed accordingly.

In the last one, you don't have a solution to the problem. You are left with a choice to either look for a new set of alternatives, or more likely, to reframe the problem, because not solving the problem was never on the table.

Given the caliber of everyone involved in these discussions, I have no doubt that alternatives y and z are already out there and just need to be hashed out like the one proposed in these discussion threads, so that we can all contrast, mix and match... etc. It does take some effort and risk to put your ideas down for others to criticize in isolation (thanks to everyone taking this leap of faith), but our mutual goals should assure you that doing so will be received with due appreciation and mutual efforts by others to do the same. This leads to contrastable alternatives, that lead to constructive discussions.

Anything less than this taking place elsewhere is something that we are lucky not to have to deal with as a group, right?

[MylesBorins]

Closing this as there has been no movement in a while, please feel free to re-open or ask me to do so if you are unable to.