Next.js: Serverless is incompatible with server-side modules that use `esm`

[Timer]

Because the esm pkg is not intended to be used with webpack, it doesn't work quite right with target: 'serverless'.

More details can be found in the original issue filed with now-cli: https://github.com/zeit/now-cli/issues/2569.

Despite the advise esm gives to prefer the module key, this is not actually feasible or a proper solution.

This mainly boils down to two reasons, broadly speaking:

1. The module field is largely a browser field, not a Node.js field.
2. Many packages are published with a CJS (main) variant that does not use the default key, meaning the non-standard switch to ESM breaks packages that work in Node.

The only viable options is to transform source code to remove esm.

[ericelliott]

Before esm existed, ECMAScript modules broke packages because babel didn't run its transforms on node_modules, so you'd get import and export syntax errors from packages you depend on. That would be fixed in Node versions that support import and export.

[Timer]

@ericelliott the module key in package.json was largely created as a webpack-specific idiom for tree-shaking front-end application code.
This key is not based on any sort of ratified standard, so there's no right or wrong answer.

The long-standing ecosystem observed behavior was to publish ES6 Modules code under the module key, and a commonjs variant under the main key.

Unfortunately, this meant some packages would end up publishing specific-for-browser versions under the module key and specific-for-Node versions under the main key.

---

The esm package is largely an attempt to solve this behavior by ensuring the main and module key are identical and isomorphic (until there's a time there's no LTS version of Node without ES6 Modules support).

However, this doesn't erase the years or months of packages who published non-isomorphic versions of packages, or didn't have main keys who exported using the .default property.

For this reason ☝️, the advice esm gives is simply not realistic (to prefer the module key for Node.js apps that are bundled).

The solution for maintaining maximum compatibility with the ecosystem and Node is to always resolve the main key, which means deleting usage of esm like https://github.com/zeit/next.js/pull/8081.

[ericelliott]

I was really trying to ask a question with my previous comment but never really asked it. The question is, does omitting esm still break dependencies with ECMAScript module exports the way it used to, because Babel ignored node_modules?

[Timer]

No @ericelliott, because they end up being bundled by webpack which converts them into commonjs.

[ericelliott]

No @ericelliott, because they end up being bundled by webpack which converts them into commonjs.

OK. So is what I'm currently doing in my package.json files providing the most ecosystem compatibility? Currently my package.json has:

{
  "main": "index.js",
  "module": "src/index.js"
}

Where main exports a standard node module via esm, and module exports a standard ECMAScript module?

The thinking is that anything relying on the legacy Node ecosystem looks at main and does the right thing for that ecosystem. Tools that understand standard ECMAScript modules look at module and do the right thing for modules.

It was mentioned above that "module" is sometimes used for code that is browser specific, but there's already a long-standing tradition that dates back to the browserify ecosystem to use a "browser" key for that, as we have used in cuid for years (because there is no universal API for crypto primitives and entropy):

{
  "browser": {
    "./lib/fingerprint.js": "./lib/fingerprint.browser.js",
    "./lib/getRandomValue.js": "./lib/getRandomValue.browser.js"
  },
}

It seems to me that using "module" to mean "browser" is a mistake and an anti-pattern that we should discourage, not enable.

❓ Do we also need a "browser-module" key? 😕

Are there any standard proposals to address these challenges, or are we all just patching around all the weird problems?

[Timer]

browser typically indicates a UMD build or similar, not a bundler-friendly build. That's why browser code is under the module key, because it spawned from webpack (first-and-foremost, a front-end bundler).

Ultimately, we have too few field choices to work with here.

For maximum ecosystem compatibility, in my experience, you must:

1. Avoid using the esm package for things intended to be used in browser and Node. This package works great for Node-only packages.
2. Ship ES Modules code, compiled down to ES5, under the modules field
3. Ship CommonJS code, compiled down to oldest non-EOL Node version, under the main field.
4. ☝️ both of these bundles should be isomorphic, working in a Node and Browser environment if you need them to (typeof window or typeof process, never process.browser).

However, more projects are moving towards compiling ./node_modules/ which allow you to ship ES.Latest under the module key. Next.js will do this soon.
Some other parts of the ecosystem still need to catch up, so this isn't generally safe yet. Do note, compiling other users code is not a safe behavior, but works in most cases.

The code shipped under the main key still should be compiled down to the oldest-non-EOL Node version.

TBH, for your specific case, I would:

1. Publish esmodules code under module key
2. Have a main field with the value of index (note no extension), this allows users to resolve .mjs first if they desire in their bundler, otherwise gives .js w/ CJS
3. Include a index.mjs that re-exports the module key's entry point (or, preferably, skip this all together since ESM in Node.js isn't a real thing)
4. Include a index.js that contains compiled commonjs code

☝️ unfortunately there's no esm in this picture since your code is designed to be used in browser and server.

Whilst this will actually be solved in Next.js soon (and you can use your current setup, with esm as-is), you may find other users / tools having to customize their setup to accommodate it.

---

Edit: Just want to add/reiterate:

The above suggestion is an opinion based on my experiencing maintaining two popular DX Toolboxes.
There is no right or wrong answer because none of these fields are based on standards.

These practices also do not reflect the direction the ecosystem is potentially heading in the future, because they're concerned with present-day _maximum compatibility_.

As an anecdote, you'll see Node 12 uses a completely different behavior and that users are beginning to ask for ES.Latest to be published.

[ericelliott]

Thank you for your patient explanation and good work. Food for thought. Ultimately, this whole incident is about the 10th time I've had to completely re-think universal module bundling in the JS ecosystem. Multiply this by a million JavaScript developers.

Note to future standards editors: Backwards compatibility matters.

[developit]

Just a note on the index.js / index.mjs thing: I like that approach and want to use it, but due to Webpack's strangely strict handling of .mjs it often ends up bundling 2 copies of the module that are separately instanced. There's a plugin that works around this (by making Webpack's behavior nonstandard).

[Timer]

This has been fixed by the new experimental-serverless-trace option released on Next.js canary and Builder canaries. It should be out in stable by 9.0.4 or 9.0.5!