Modules: Next steps after feature discovery: Proposed solutions + User Journeys

[MylesBorins]

So it seems like we are getting close to the end of feature discovery and this is leading to some deep conversations around implementation... which we definitely need to have but I think we are perhaps skipping a step.

I'd like to propose that for the next stage we focus on creating proposed solutions that tie together a selection of the discovered features and pair that with an example user journey of how individuals will be able to transition existing ecosystem + project code to the proposed solution.

A brief example built on https://github.com/nodejs/modules/issues/82

Implementation

• plugin system for loaders is introduced
  
  
  
  • default loader uses the package-name-maps proposal for resolution
  
  
  • default loader is designed to be 100% compatible with the platform (browser + node interop)
  
  
  • additional "official" loaders can be packaged in node as opt in
  
  
  • classic loader
    
    
    
    
    
    • "bundled" loader combining json, path crawling, native modules, etc.
    
    
    
    • each of these features can be individually opted in to
    
    
    
  
  
  • as new goals are introduced to the platform they are support by the default loader
  
  
  • wasm can start as an "official" non default loader and be merged into default loader when supported by platform
  
  
  • e.g. we can work with standards to try and support importing of JSON ()
  
  
  • loaders can be "bundled" to single meta loader
  
  
  • loaders will define a mapping between file extensions & a custom loader
  
  
  • loaders are top-level, only supported at the application level
  
  
  • non default loaders can be specified in the package json via a "loaders" field and cascade
  
  
  • "loaders": ["@nodejs/classic"]
  
  
  • "loaders": ["@std/esm"]
  
  
  • "loaders": ["@typescript/loader", "@nodejs/loaders/native-modules", "@nodejs/loaders/json"]
  
  
  • "loaders": ["@mycompany/loader"] # same as above 😉
  
  
  • loader meta data can be bootstrapped on by transpilers and bundlers such as babel + webpack to allow for minimal configuration 💯
  
  
• import.meta.require can be used to escape hatch into cjs without requiring a custom loader
  
  
  
  • transpilers can handle this out of the box without referencing a custom loader to create browser compatible esm or script
  
  

transition user story

The Company has a large code base that is currently transpiling using babel. They are using import for both esm and cjs. They want to start using esm natively in their code base without transpilation

1) move to node.js esm implementation but cascade the @std/esm or @babel/loader custom loader to simplify things at first
2) transpile current app code using babel + babel loader to esm code, converting all current import 'cjs-thing' to import.meta.require without requiring direct user intervention.
3) test transpiled code and confirm it works as expected
4) remove custom loaders from package.json
5) as dependencies move from cjs -> esm update the import.meta.require statements to import

---

This is not complete by any means but an attempt to illustrate how I think we can approach this.

Open to any thoughts or criticism, please lmk

[GeoffreyBooth]

There's a lot I like about this proposal, but is this thread about the proposal or about next steps?

Re next steps, I thought we were going to create issues for each feature. We have a long way to go on that. Should people just create issues from the features Google doc? Maybe each person could open an issue for a feature they feel knowledgeable about and can describe well?

[MylesBorins]

@GeoffreyBooth seems I may have missed a bit from the last meeting. With the amount of discussion around implementation I thought this approach might be a better way to move forward. If I in turn am missing a step I'm more than happy to close this for now and bring it back when appropriate.

Regarding the proposal I included, I started writing something small to use as an example and it just kind of took off... not expecting this thread to be used to discuss it. Really glad you like it though :smile:. My plan would be to open another thread specifically for this proposal and polish it further with a more defined implementation are more user journeys for transition.

To be explicit, I'd prefer we don't discuss this proposal in the thread, I'll move the text over to my other issue.

[guybedford]

Firstly, it's really great to see all the discussion happening and that people are engaging in the various threads. This show's that we're all tackling the problems, even if the solutions may still seem far off it is a huge step in the right direction.

In terms of process how about instead of moving straight to proposals, that we look at our features and aim to group them and then categorize them on two axes - priority and proposal status.

For priority, some gradient along the lines of:

1. Definitely for unflagged modules release
2. possibly for unflagged modules release
3. Definitely after unflagged modules release
4. Possibly after unflagged modules release

Then for proposal status something like:

a) What is currently supported
b) What has an existing proposal
c) What seems like it has an approach but would need to still be fleshed out into a proposal
d) Something where further research is necessary to determine an approach

Further thoughts / ideas welcome here in terms of trying to work out how to distinguish low-hanging fruit and easy wins while maintaining eyes on the important balls.

[guybedford]

Perhaps also to clearly allow uncertain distinctions on the above as well if it isn't clear under what status and priority a feature falls under.

[GeoffreyBooth]

@benjamingr wrote in https://github.com/nodejs/modules/issues/115#issuecomment-390469882 (which I’m migrating here to avoid polluting that thread):

As a maybe silly side note - all these issues are getting a little overwhelming. I want to read them all and think about them to make better choices when voting on things but there is a lot of them at once.

I hear you. I was just following up on our action item from the last meeting, where we discussed the next step as being creating issues for each feature in the features document. I tried to honor the intent of each requested feature, as much as I could discern it from rereading the referenced use cases. I definitely think there’s room for grooming these features, for improving the descriptions of many of them and for splitting or combining others. Please feel free to suggest improvements. I was trying to follow the features document as best I could, to avoid neglecting someone’s request, though for some features with identical use cases I couldn’t figure out how one feature was different from the other and so I combined them. That said, some features like #100 are so huge that I think we’d be better off splitting them up and closing the parent. I’m open to feedback.

I wanted to get this done now because people were already starting to dive into implementations, and I didn’t want the features to get forgotten in that discussion. I think it’s easier to discuss implementations’ limitations when there are features/issues to directly link to.

Note also that there are still a few more use cases that haven’t yet gotten assigned into features, so we’re not done adding features yet. We also need features for the spec itself: what is ESM? Most of the use cases and features we have assume a baseline of “import and export statements work according to the ESM spec” but we don’t really have features defining that. An implementation that satisfies most of our current features _could_ be something as ridiculous as “integrate Babel into the Node runtime and transpile all import and export statements into CommonJS before the code gets evaluated” but I’m assuming that wouldn’t fulfill plenty of ESM spec-related requirements. It would be good to have those listed too so that they can be part of the evaluation process.

[robpalme]

This issue brings to light something I have not previously seen discussed
properly: the concept of use-cases/features being "default" vs "possible"
vs "unsupported".

I'd suggest we should revisit the use-cases and classify each as either
"needed by default" vs "needs to be possible". This will allow proposed
solutions to be assessed in a more precise manner.

On Sun, 20 May 2018, 12:58 Geoffrey Booth, [e-mail] wrote:

@benjamingr https://github.com/benjamingr wrote in #115 (comment)
https://github.com/nodejs/modules/issues/115#issuecomment-390469882
(which I’m migrating here to avoid polluting that thread):

As a maybe silly side note - all these issues are getting a little
overwhelming. I want to read them all and think about them to make better
choices when voting on things but there is a lot of them at once.

I hear you. I was just following up on our action item from the last
meeting, where we discussed the next step as being creating issues for each
feature in the features document
https://docs.google.com/document/d/1HuH5E1d_mNuziBxPDRZVjpdfmUPhqVg0MUUcYDnqwn8/edit#.
I tried to honor the intent of each requested feature, as much as I could
discern it from rereading the referenced use cases. I definitely think
there’s room for grooming these features, for improving the descriptions of
many of them and for splitting or combining others. Please feel free to
suggest improvements. I was trying to follow the features document as best
I could, to avoid neglecting someone’s request, though for some features
with identical use cases I couldn’t figure out how one feature was
different from the other and so I combined them. That said, some features
like #100 https://github.com/nodejs/modules/issues/100 are so huge that
I think we’d be better off splitting them up and closing the parent. I’m
open to feedback.

I wanted to get this done now because people were already starting to dive
into implementations, and I didn’t want the features to get forgotten in
that discussion. I think it’s easier to discuss implementations’
limitations when there are features/issues to directly link to.

Note also that there are still a few more use cases that haven’t yet
gotten assigned into features, so we’re not done adding features yet. We
also need features for the spec itself: what is ESM? Most of the use cases
and features we have assume a baseline of “import and export statements
work according to the ESM spec” but we don’t really have features defining
that. An implementation that satisfies most of our current features
could be something as ridiculous as “integrate Babel into the Node
runtime and transpile all import and export statements into CommonJS
before the code gets evaluated” but I’m assuming that wouldn’t fulfill
plenty of ESM spec-related requirements. It would be good to have those
listed too so that they can be part of the evaluation process.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/86#issuecomment-390495951, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGni9cAqpo4Q3ohesqRo5XVILaSG3bedks5t0aCggaJpZM4UCehd
.

[ljharb]

Absolutely - some features will be necessary as defaults by many, but it’s likely that all features can be possible “somehow” (loader hooks, etc) - the primary issue at hand is determining which features must be default.

[guybedford]

Should we mark this issue for the agenda this week?

[MylesBorins]

Sgtm

On Sun, Jun 3, 2018, 12:14 PM Guy Bedford [e-mail] wrote:

Should we mark this issue for the agenda this week?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/86#issuecomment-394151632, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecV7uNKFGOIBW7B-m0wOZiRevWUnvXks5t47ccgaJpZM4UCehd
.

[GeoffreyBooth]

@MylesBorins At the last meeting you mentioned that a priority for you was browser equivalence. Do you mind elaborating on that, perhaps as a user story and/or some features?

Others such as @bmeck have mentioned how important spec compliance is for them, but I don’t see that reflected in our user stories or features. Perhaps they could add some user stories and features along these lines? Ideally there wouldn’t be just one huge feature called “spec compliance,” but rather it could be broken out into digestible chunks.

Not trying to prolong the feature discovery process, but I want everyone’s priorities to be represented in our features list and how we evaluate potential solutions.

[MylesBorins]

Going to close this for now.