Modules: Guiding Design Principles

[MylesBorins]

The TSC released a medium post outlining the state of ESM as well as some high level design principals

• We are committed to shipping ESM
• We are committed to ESM being first class in Node.js
• We are committed to having the Node.js and Web platform as first class runtimes for modules.
• Modules installed via npm should be able to run after installation without requiring a build step.
• We are committed to supporting our current users and offering migration paths as necessary. This can be through interoperability or APIs to improve the developer experience of working between module systems.

As a new working group I think it will be important for us to make sure these principles align with our groups vision. If you are open to adopting the TSC's vision, please simply use a 👍🏽 reaction. If you would like to see changes please use this thread to offer opinions

[guybedford]

I am in full agreement with all these goals 👍🏻

In particular “modules installed via npm should be able to run after
installation without a build step” is a goal I’ve been pursuing for some
time through my own tooling work, and was why I created SystemJS which
provides support for this through dynamic require tracing in the browser.

That said I think it could do with some clarification from the perspective
of this group for the following reasons:

• The only way to load commonjs in the browser without a build step would
  be to (a) use dynamic require tracing in the browser which means custom
  source parsing in-browser, and (b) have the browser run resolution by
  hitting 404s on the node module lookup algorithm. (a) has a performance
  penalty and doesn’t cover all dynamic require cases like requiring
  identifiers. (b) has a performance penalty of latency which adds up
  surprisingly quickly as well.
• So having written a project that can do (a) and having played with
  alternatives to (b) seeing all the edge cases play out, I just don’t feel
  this is something a group like this could recommend as it feels like a
  fragile approach.
• This does not mean I don’t agree with the goal overall though, but I’d
  hope we can reframe this goal in a way that is perfectly solvable by this
  group.

Perhaps we can put some thought to how we can describe the browser
requirements in a flexible way that covers these goals without writing
implementations into a corner, including:

• Accepting that there will be many approaches to getting commonjs modules
  to work in the browser, and that some userland experimentation will be
  required to find the best approach for Es modules workflows. There can be
  approaches with service workers, compiling servers, precompile steps after
  npm install and in future browser hooks. We should embrace the flexibility
  here.
• Make sure that the goals of these tools are taken into account in design
  decisions - this is crucial. But ideally in a way that does not dictate the
  way such tools should be implemented.
• Setting the goal of this group to ensure we can define edge cases and
  interop so all those tools can agree on the behaviours and users can trust
  their code to work between tools. This allows specification to do what it
  does best in allowing the whole ecosystem to move forward with trusted
  conventions regardless of exact toolchain. For example service-worker-based
  non-build dev workflows can evolve into browser hooks dev workflows
  following the same conventions. This would be a huge benefit allowing and
  enabling creativity for these approaches in the ecosystem while not
  constraining the solutions.

Let me know what you all think. Perhaps we can split this into its own
thread too.

On Tue, 06 Feb 2018 at 00:25, Myles Borins [e-mail] wrote:

The TSC released a medium post
https://medium.com/the-node-js-collection/the-current-state-of-implementation-and-planning-for-esmodules-a4ecb2aac07a
outlining the state of ESM as well as some high level design principals

• We are committed to shipping ESM
• We are committed to ESM being first class in Node.js
• We are committed to having the Node.js and Web platform as first
  class runtimes for modules. * Modules installed via npm should be able to
  run after installation without requiring a build step.
• We are committed to supporting our current users and offering
  migration paths as necessary. This can be through interoperability or APIs
  to improve the developer experience of working between module systems.

As a new working group I think it will be important for us to make sure
these principles align with our groups vision. If you are open to adopting
the TSC's vision, please simply use a 👍🏽 reaction. If you would like to
see changes please use this thread to offer opinions

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/11, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkiylDgQoV8x-mMG-y28Cxt_Y2hfbWLks5tR3_HgaJpZM4R6Meo
.

[mcollina]

@guybedford I do not understand your last comment, and how you mention loading cjs in the browser in this situation (and how it relates to the "Guiding Design Principles"). As far as understand it, that is not part of this team activity, nor something we want to address in Node.js itself.

The only part that is related to the browser is that we would like our ESM implementation to be fully compatible, and ESM that do not access any specific Node.js things should work on both. That does not include cjs.

[ljharb]

To clarify, "modules installed via npm should be able to run after installation without a build step" to me means "no postinstall script", not "no prepublish script".

[guybedford]

@mcollina your question touches on a lot of topics... if you don't mind me avoiding the long answer for now and instead allow me to provide an example, note that supporting import.meta.require in ES modules in the browser is equivalent to supporting CJS modules in the browser. It is at these sorts of levels that these concerns touch.

@ljharb right, that would be a useful clarification, but the lines of my argument above is that we shouldn't attempt to impose such a form on workflows unless we are absolutely sure we can get consensus on such an approach for all users and all workflows.

[mcollina]

I would classify import.meta.require as a "node specific thing" for the time being.

[guybedford]

@mcollina it sounds like you're advocating removing the fourth bullet point altogether then!

[mcollina]

it sounds like you're advocating removing the fourth bullet point altogether then!

Absolutely not, in fact it's the exact contrary.
If you want full compatibility with the Web Platform without transpiling and/or some runtime support, your module and its dependencies needs to be ESM and not to be using any node-specific API or modules (import.meta.require would just be such an API - if we ship it). This enable the community forces to migrate the packages to ESM in their own time.
This is a much smaller goal than supporting require inside the browsers.

[guybedford]

@mcollina so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

My argument is to (1) maintain the scope of the fourth point to include loading CommonJS in the browser, but (2) as opposed to trying to provide a specific solution for this, to reduce the scope to treating this as an understood use case that things like interop and loading conventions must bear in mind, while not dictating what those exact workflow solutions must be.

I think if we choose to ignore browser workflows for CommonJS interop entirely, that would be disregarding many interesting use cases. We don't need to know the answers, we just need to appreciate the questions.

[mcollina]

so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

Not exactly. I would articulate it better something like the following:

1. CJS and ESM installed via npm should be able to run _in Node.js_ after installation without requiring a build step - I believe this is the original meaning of the design principle as formulated by the TSC.
2. ESM installed via npm should be able to run _in the Browser_ after installation without requiring a build step if it does not use any Node.js specific API.

Having a solution for cjs interop in the browser would be very nice. However, that would almost necessarily require some pre-processing and a runtime library. I would rather leave this problem to other groups (webpack, browserify, rollup etc..) to tackle.

[guybedford]

Ahh, I've been going by the Medium post definition of point 4 which specifically names the Web Platform here, which may explain the confusion:

_We are committed to having the Node.js and Web platform as first class runtimes for modules. Modules installed via npm should be able to run after installation without requiring a build step._

@mcollina I think we're mostly in agreement actually, although while not directly tackling the browser interop problems, I hope we can keep those workflows in mind here that's all.

Browser unification will keep coming up in discussion here so I'm just trying to consider how we can frame those arguments here.... to be able to consider and appreciate the different types of browser workflows, and allow their consideration to affect decision making (eg if we find ourselves considering something that makes service worker workflows impossible, or static builds impossible, that should be a valid consideration), without trying to impose solutions or decide on exactly how we think those workflows should work (assuming that everyone will do post-compile on npm install for example, and ignoring all other possibilities).

[giltayar]

@guybedford - I believe the npm proposal touches on a solution to your problem: given that ESM definitions are static, theoretically npm install can go over all imports inside the installed modules and statically link them to their exact file, e.g. from import 'D' to import '../node_modules/D/index.mjs, thus enabling browsers to directly access the correct file. In essence, changing all bare imports to relative imports that can be understood by the browsers.

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports. Maybe, just maybe, we can solicit the help of the browser vendors and programatically supply a map from bare imports to "real paths".

So, yes, without touching the files, and without the browser vendors help, there is no complete solution, but given that npm (and yarn) are perceived as part of the solution, and given that the browsers are part of the problem :-), maybe we can ask for their help.

[guybedford]

@giltayar

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports.

This is exactly the sort of mindset I'm trying to reframe against - we cannot say we have "solved" something and then admit it is only "part of the solution". Anything which is only "part of the solution" must change over time to become a "whole solution". And so the very solution may well be something else entirely.

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration. So we shouldn't assume that it is an npm install process that does post-processing, as this approach hasn't been proven in the wild. For example linking workflows are supposed to be first class in npm, yet break down with this.

[devsnek]

if the web ever gets custom loaders npm or node could just host a little script on a cdn for a custom loader that uses node's resolve algorithm. seems pretty painless considering how annoying it is that people expect node's resolver pattern to be picked up by browser :/

[bmeck]

@devsnek you can implement custom Loaders with a proxy algorithm in a http server or service worker already (except for data: and blob: URLs, which are their own special case since they can't do relative imports). We ended up making sure these work before moving forward with designs on per package hooks.

[evanplaice]

Here's an attempt to recap and surface some of the identified issues.

1. The term modules should specifically mean ESM as they are specified in ES262

• Link to ES262-15 Scripts and Modules

2. CJS != ESM, and CJS browser interop falls outside the scope of this group

• CJS interop will be provided in some way, shape, or form in Node.js only
• CJS in the browser falls outside the scope of this group.

3. import.meta, is the spec-compliant location to put platform-specific implementations

Assuming spec-compliance is the intended long-term goal to ensure node/browser interop, import.meta should become the canonical place to define platform-specific behavior.

• in the short term, ensure backward compatibility by leaving existing built-ins unchanged
• in the short term, provide aliases to the existing built-ins as a gradual upgrade path
• in the mid-term, introduce deprecation messages for built-in usage outside of import.meta
• in the long-term provide an adapter to allow backward compatibility
• in the long-term, remove existing built-ins, import.meta becomes the default built-in

The upgrade path/transition to import.meta is an intended goal. It is a direction we are working toward, it may not actually be the final end goal. If it's unrealistic to remove the existing built-ins without breaking the NPM ecosystem they could be left indefinitely. Instead of axe-grinding over the many possible futures, lets focus on what we know and don't know.

Here's a good place to start:

As it currently stands. Import.meta is only at Stage 3. We're a bit ahead of the curve so it would be beneficial to reach out to TC39 to see what the hold up is.

Update: The import.meta is good to go, just awaiting an alternative implementation as proof before Stage 4.

4. Bare imports are not specified, this needs to be addressed

Update: Implementation specifics that address how bare imports are handled in browsers don't fit within the scope of the principles mentioned in this issue. Feel free to ignore this item.

If we follow the ESM spec as it currently stands, that means imports require direct links to the source (ex node_modules ) with the extension. I don't think I have to spell out why that would suck.

Here are the facts:

• there is no existing or proposed spec for handling bare imports
• inventing an approach may introduce interop issues in the future
• ESM support in browsers is too bleeding edge to make a solid determination at this time
• npm asset is coming soon, direct imports won't be so bad for non node_modules
• the node_modules should provide a 1:1 equivalent to the current CJS approach

If we're going to push browsers to provide a process for handling bare imports it should be:

• clearly specified and submitted as a new TC39 proposal
• may change throughout the spec process
• may take considerable time to push through to full specification

If we want to set the standard, it should be done right. Else, it has the potential to cause issues down the road if a future spec is defined that diverges from the Node-specific implementation.

5. An ESM-first approach to ESM->CJS interop may not be possible without top-level async

This topic is too complex to summarize here. See Module Interop

[giltayar]

@guybedford

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration.

Exactly! My attempt to paint a solution may have been misguided, or not clear enough, but I was trying to argue against your following sentence:

The only way to load commonjs in the browser without a build step would
be to (a) use dynamic require tracing in the browser which means custom
source parsing in-browser, and (b) have the browser run resolution by
hitting 404s on the node module lookup algorithm.

The funny thing is, now that I reread the sentence, is that I missed the word "commonjs" in there, and read it as "the only way to load esm in the browser without a build step...". I was trying to propose a solution _for ESM_.

So sorry about that, but the good thing is that it seems that we are in agreement that there _can_ be a solution for using NodeJS ESM modules in the browser (that do not rely on NodeJS stuff) without a build step. (I think...)

Just to understand - if you don't believe an npm install rewrite is the direction to a solution, how do you believe this should work?

[bmeck]

@evanplaice

1. It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

2. Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

• npm asset is coming soon, direct imports won't be so bad for non node_modules

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

• the node_modules should provide a 1:1 equivalent to the current CJS approach

This is not entirely possible for a variety of reasons such as Errors being permanent, moving to URL based specifiers, etc.

In summation, TC39 is not the place to put resolution.

[evanplaice]

It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

Copy. So, awaiting implementation proof. There's no axe-grinding over implementation specifics.

Update: Sorry about any confusion, I went back and removed any comments/statements not within the scope of this issue.

Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

Agreed

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

Disregard any mention of npm asset. It's outside the scope of this group.

[XadillaX]

How about do require to import an ESM with export default?

[giltayar]

@mcollina made more precise one of the design guidelines:

ESM installed via npm should be able to run _in the Browser_ after installation without requiring a build step if it does not use any Node.js specific API.

I'm assuming something like npm asset does _not_ comply with this guideline, right? If somebody thinks it does comply with the guideline, please say so.

We've been discussing npm assets here, and, assuming it does not comply with the guideline, then I believe it should be out of scope of this group.

[bmeck]

@XadillaX unfortunately that is not possible due to some timing problems with ESM only working synchronously when no other module formats are involved. There are a few old slides on this matter. In essence, ESM must be loaded asynchronously.

@giltayar Since it is an explicit pre-processing step that changes source text I would agree. I can't imagine things like subresource integrity style workflows working when the source text being verified is altered by a preprocessing step potentially per install.

[giltayar]

@bmeck - I agree. Which brings me to the next question: does the working group believe that userland service workers (or any userland mechanism in the browser) is also a mechanism that is complying with the same guideline I discussed (and I will requote it again here just for reference)?

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

I think not. A solution in terms of service workers that the developer writes doing some kind of module resolution similar to what NodeJS does, IMHO, does _not_ comply with this requirement.

[bmeck]

@giltayar I am unclear on that one, with pathing and format mutation servers also can do things at the same level as service workers. To note; APMs, Mocking, transpilers, etc. all require hooks that would be able to provide these redirection or source text mutation behavior, but will not be available in the browser. There might be some level of conflict that cannot be resolved between these 2 points. I think we can discuss this further if hooks should be excluded from this goal.

[mcollina]

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

I think not. A solution in terms of service workers that the developer writes doing some kind of module resolution similar to what NodeJS does, IMHO, does not comply with this requirement.

As I said:

1. CJS and ESM installed via npm should be able to run in Node.js after installation without requiring a build step - __I believe this is the original meaning of the design principle as formulated by the TSC__.
2. ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API. _This is __my__ interpretation_.

We can go back and ask the TSC about this. Or we can come up with our own recommendation cc @MylesBorins. I'm personally _fine_ with npm asset.

[devsnek]

imo misc package managers shouldn't be part of our solution. node's core should be able to meet our expectations by itself.

[jdalton]

The guiding principles seems a bit ESM heavy for a general modules group. I'd like CJS to get a call out for continued support. I'd also like to reduce the us vs. them mindset. Node would not be what it is today without npm and the ecosystem it enabled. I'd like developer experience and being pragmatic to be a guiding principle.

[MylesBorins]

fwiw this group was originally spun up specifically to deal with ESM and
interop... perhaps the name "modules" is a bit too generic.

Scoping this seems like a pretty good early step, especially as we figure
out what we want to be responsible for when chartered... I personally think
this group taking over the cjs implementation is unnecessary

On Wed, Feb 7, 2018 at 12:55 PM John-David Dalton [e-mail]
wrote:

The guiding principles seems a bit ESM heavy for a general modules group.
I'd like CJS to get a call out for continued support. I'd also like to
reduce the us vs. them mindset. Node would not be what it is today without
npm and the ecosystem it enabled. I'd like developer experience and being
pragmatic to be a guiding principle.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/11#issuecomment-363854137, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecVxS0N1AxwtPLSijIi4dUKooPPvYdks5tSeOrgaJpZM4R6Meo
.

[jdalton]

Coo coo. I thought, through discussion, the group scope was widened to more than ESM. The name was originally ESM centric then made generic. I hope my other comments aren't lost in the shuffle though.

[bmeck]

@jdalton has a very good point where we shouldn't entirely leave CJS out of the picture. If we introduce new APIs like the --loader hooks we should probably see if they can be placed in CJS as well without issues / if the issues are big enough to stop having a single workflow in both module systems.

[MylesBorins]

for the record, I am open to expanding the scope. This should be a top agenda item in our first meeting

I've opened https://github.com/nodejs/modules/issues/17 to further discuss

[giltayar]

@mcollina

I'm personally fine with npm asset.

Isn't that in conflict with your interpretation of the guiding principle (which I totally agree with!):

ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API

npm asset is a build step, is it not? And if it is, we shouldn't require it after running npm install. the ESM modules should "just work" in the browser, without the need for npm asset.

[mcollina]

@giltayar Yes, definitely. I hope I clarified how I read the principles above and what is my opinion on how we should treat the browser issue: it might not be possible to achieve, and we’d have to compromise and/or propose changes to TC39.

[WebReflection]

@mcollina as previously mentioned I think we can simplify our requirements into:

• ESM that already runs in compatible browsers should run in NodeJS without requiring a build step

This covers the way people publish modules, with or without convenience for NodeJS (such as asset).

Worth mentioning, since I've seen already sneaky arguments, that _works in browsers_ means with static servers, not with dynamic resolution dependency on the server to find out which file was required.

So maybe we could agree on a design principle that will not directly affect the already existing ESM ecosystem.

[mcollina]

@WebReflection please stop arguing for the .js extension. You are sneaking that topic here and in another issue, and none of them are the place to discuss that. If you want Node to revisit that decision, then open a new issue.

[giltayar]

I've just opened #18 to deal with the issue of browser/node compatibility, so we can move the discussion about this to there.

[WebReflection]

@mcollina there's no mention to any extension but it's kinda of implicit part of that "_modules installed via npm should be able to run after installation without requiring a build step_" point.

[ljharb]

I read that as, "without requiring a post-installation build step" - eg, it's totally fine to require a pre-installation build step, like prepublish.

Perhaps that point's language should be clarified to remove the ambiguity?

[MylesBorins]

When written my intention was that it meant "without a development or production build step".

[ljharb]

Gotcha, thanks for clarifying.

Regardless, since you can hand-write a .mjs file as easily as a .js file, I don't think that the extension is relevant to a "build step" criteria.

[bmeck]

@MylesBorins are we still needing to do stuff outside of the goals PR here?

[MylesBorins]

I'm going to go ahead and close this. We can open a fresh thread if we need to revisit with current conversation