Modules: The type field and `.node`

[jkrems]

So far we've told people that type can be used to switch how .js is interpreted. In https://github.com/nodejs/node/pull/31388 @bmeck pointed out that setting it to module also has side-effects on how .node is interpreted. To me that sounds really counter-intuitive and surprising. Are there any meeting notes for this? Right now I would consider it a bug that's worth fixing.

For reference, this is the current effective extension map delta:

const typeModule = {
  '': 'module',
  '.js': 'module',
  '.node': 'error',
};

const typeCommonJS = {
  '': 'commonjs',
  '.js': 'commonjs',
  '.node': 'commonjs',
};

I'm omitting .json because with experimental JSON modules the difference goes away. But without the flag, it is also affected by type.

[bmeck]

If memory serves it was pulled out due to compat concerns. Don't remember when, but seems we should discuss what type "module" is intended for rather than adding .node to it prematurely

[ljharb]

I’ll again link to #283 as the imo preferred way to handle this ambiguity that will come up for an unknowable number of new module types in the future.

[jkrems]

If memory serves it was pulled out due to compat concerns.

I think my memory was that we intended for .node to always be an error when imported, not just when it happens to be a type:module package. But it's also totally possible that I misunderstood the discussion when it happened.

I’ll again link to #283 as the imo preferred way to handle this ambiguity that will come up for an unknowable number of new module types in the future.

If node doesn't support a new extension, then it doesn't seem to be valuable for a package to declare it. If a loader hook adds the support for loading a format, it can also provide the format. Allowing packages to come up with their novelty extensions for existing formats doesn't seem like a super compelling use case to me..?

Our current direction for loader hooks (determined before loading user code, immutable once set) seem to preclude that the package's extension map can introduce new formats.

[GeoffreyBooth]

We have two categories of files: those that have an explicit parse goal inherent in the file type, and ambiguous files that can be either Script or Module. .mjs and .cjs are examples of the former, .js is the latter. How would people categorize all of our other natively supported types: .node, .json, .wasm?

My understanding, please correct me:

• WASM is always Module, so equivalent to .mjs.
• JSON can be either, so equivalent to .js; but it’s always the same in either parse goal so the parse goal/"type" is irrelevant.
• .node . . . ?

I think my memory was that we intended for .node to _always_ be an error when imported

Yes, this is my recollection as well. We document in https://nodejs.org/api/esm.html#esm_commonjs_json_and_native_modules that JSON and native modules need to be pulled into an ES module via require (from createRequire).

[ljharb]

@jkrems it's not about allowing novelty extensions (altho ofc it would allow that), it's about not having to decide the semantics of a field like "type" for all future extensions as well as the complex combo of current ones.

The extension map would be a subset of loader hooks; they don't conflict as far as I'm aware.

[bmeck]

@GeoffreyBooth

How would people categorize all of our other natively supported types: .node, .json, .wasm? ...

I'd like to be wary of attempting to see if something "is a Module" (unclear if meaning an Abstract Module Record here or in "type":"module"). WASM can be loaded through import but also just via an API. Virtually anything can be transformed into an Abstract Module Record and many things can be exposed without using Module Records at all (JSON/WASM/etc.).

I don't fully understand the comment.

[jkrems]

it's about not having to decide the semantics of a field like "type" for all future extensions as well as the complex combo of current ones.

To me that comes down to the question: Do we believe that we'll add more ambiguous file extensions? As long as extension-less and .js are the only ambiguous file extensions, it feels perfectly fine to say "type determines how to interpret those". I would hope that we never add another, distinct, new kind of ambiguous file extension to node.

[jkrems]

JSON can be either

Somewhat echoing @bmeck: No, it can be neither. "module" refers to ESM specifically, not any kind of entry in either "module map"/cache. JSON is always application/json, independent of require vs. import.

[bmeck]

@jkrems at least in my last call with Dan they wanted WASI and WASM to be in the same file extension.

[GeoffreyBooth]

Don't remember when, but seems we should discuss what type "module" is intended for rather than adding .node to it prematurely

The "type" field is intended to define the parse goal of ambiguous files when they’re referenced via import. That’s why it applies to both .js and extensionless files.

We only need to worry about how "type" affects other file types when we want to add support for them to import _and they’re supported in require._ Assuming we never support require of WASM, then import can always import WASM as an ES module (e.g. using the Module parse goal) and that’s it.

I'd like to be wary of attempting to see if something "is a Module" (unclear if meaning an Abstract Module Record here or in "type":"module").

I’m referring to parse goal. I thought that all WASM files were in parse goal Module, are they not? If so, .wasm is equivalent to .mjs in that regard. To be used in a CommonJS context, one would need to use import().

[bmeck]

I thought that all WASM files were in parse goal Module, are they not?

They are not. They are a completely different parser.

[jkrems]

Dan they wanted WASI and WASM to be in the same file extension.

That feels a bit awkward for node. I guess it means that node wouldn't support "raw" WASM imports, only ones with WASI semantics (including capabilities..?). For the former there's always the direct APIs though and fortunately we still have a dedicated experimental flag for importing .wasm.

[GeoffreyBooth]

I thought that all WASM files were in parse goal Module, are they not?

They are not. They are a completely different parser.

How will browsers support loading WASM? Via import statements? Will they also support WASM via <script> tags, and if so, what would the type of such a tag be?

I assume WASM has only one parse goal?

[bmeck]

So it sounds like browsers are treated WASM the same way they treat ES module JavaScript.

No, they remain decoupled in terms of format. They use the same loader workflow though.

Is there an issue with Node treating .wasm files the same way we treat .mjs files?

Yes, that would not allow WASM to be loaded. I don't think I understand this question fully.

In that they can only be used via import/import()

This is not true today, per the WebAssembly global.

and they are always parsed the same way regardless of "type".

This likely won't be possible if WASI/WASM fully diverge.

[jkrems]

Is there an issue with Node treating .wasm files the same way we treat .mjs files? In that they can only be used via import/`import(), and they are always parsed the same way regardless of "type".

That's already true behind --experimental-modules. But doesn't address extension-less files.

Will they also support WASM via <script> tags, and if so, what would the type of such a tag be?

I think one conceptual issue here is that the type field in package.json and the type attribute in the <script> tag are fundamentally different. The field in package.json determines the Content-Type of ambiguous files. It has no effect on the module system. The attribute on <script> tags determines the module system to use (or rather to enable a module system at all). It has no effect on the Content-Type.

With this, the type attribute of a <script> tag that loads WASM is clearly "module" because it's loaded using the module system. But that doesn't tell us anything about the package.json field because the Content-Type header is what determines the file format. And the file format is what is controlled by the package.json field.

[bmeck]

The field in package.json determines the Content-Type of ambiguous files.

This is not purely the case, it does have potential alterations to other things like which loader to use for the main entry point even if the file of the main entry point is not "ambiguous".

[GeoffreyBooth]

@bmeck In all cases I’m just talking about the public API that end users see, with regard to importing/loading the file. It sounds like .wasm is equivalent to .mjs as far as end users are concerned (both are imported via the script type=module, import, etc.).

[bmeck]

It sounds like .wasm is equivalent to .mjs as far as end users are concerned (both are imported via the script type=module, import, etc.).

I don't understand this. Not for/against any part of the comment; I simply am confused.

[GeoffreyBooth]

I don't understand this. Not for/against any part of the comment; I simply am confused.

End users don’t really need to know or care about the module system or what’s happening under the hood. They can just treat .wasm files the same way they treat ES module JavaScript files, as far as script tags and import statements are concerned. I assume the server needs to send a WASM Content-Type, which is equivalent to us requiring a .wasm extension.

[bmeck]

discussion seems to be focused around WASM after some minor back and forth; maybe we can migrate the last bit of discussion to https://github.com/nodejs/modules/issues/457 ; feel free to reopen if there is disagreement