Modules: Revisit dot-main in exports field

[jkrems]

Originally the exports proposal had a way to specify the main field as well. The following was the equivalent of setting main to './lib/entry.mjs':

{
  "exports": {
    ".": "./lib/entry.mjs"
  }
}

We discarded this for two reasons:

1. Without dual-mode, having two separate values seemed confusing.
2. There were educational and aesthetic concerns with a property key that's just "." without additional context.

But with fallbacks and/or differential serving, we run into a new reason for wanting this second key: Backwards compatibility.

The problem is that if I'm maintaining a package std-x-polyfill and I want to fall back to the native std:x module where it's available, I could express it like this:

{
  "main": ["std:x", "./lib/x-polyfill.cjs"]
}

The problem is now: This package cannot be used in anything but the latest version of node. Older versions will not recognize this kind of main field. If we'd support exports[.] as an alternative with higher priority, the problem can be solved (if a bit verbose):

{
  "main": "./lib/x-polyfill.cjs",
  "exports": {
    ".": ["std:x", "./lib/x-polyfill.cjs"]
  }
}

[ljharb]

i'm not aware of any proposal that would ever alter how "main" works, including adding array support.

What I'd expect is something like this:

{
  "main": "./path",
  "exports": {
    "./path": ["std:x", "./lib/x-polyfill.cjs"]
  }
}

with a path.js as a backwards-compat fallback.

[guybedford]

Supporting relative ./paths on the LHS in exports is a sort of internal rewriting system.

The concern with these approaches is that you can import an absolute file path, but not necessarily actually get it when you load it! So there is a lack of transparency to the user.

Contrast this with exports, where the package name boundary naturally forms the encapsulation, instead of it applying to all types of importing. Users already know there is some indirection with package imports as import 'pkg' will import the main, so this sort of naturally extends to the package subpaths when importing the bare specifier. While importing /path/to/file.js suddenly loading another file is quite a different thing.

The point being, if we want to do "internal rewrites" that is a decision I think we should make very carefully thinking about the usability implications, and justified based on its unique merits.

[jkrems]

@ljharb Your solution seems to leak the existence of the polyfill onto the exports while also assuming that exports are something happening after main, adding one more level of indirection. I think both are surprising. Exports, unlike browser, is not an environment-specific override. It's a first-class mapping, just like main. So it applying after main seems counterintuitive.

[ljharb]

Right - i see it as more of a virtual filesystem, which to me “main” sits on top of - iow i do not see main and exports as siblings, but rather “main” as a high level thing and “exports” as a low level thing.

[jkrems]

I'm not sure I follow how main is any different here.main maps "foo" to an internal path, exports maps "foo/bar" to an internal path. The only difference is the substring after "foo" which is "" for main.

If we start asking people to add intermediate junk paths to exports so they can use its features for main, it seems like we're actively undoing the clean interface main+exports could provide. Now I suddenly need to support both "foo" and "foo/<some-opaque-id>" as my interface..?

[ljharb]

Maybe I’m not clear on what you want either. I don’t think the format of “main” should ever change. What exact goal do you want to achieve that couldn’t be done with an unchanging main format?

[jkrems]

The use case is in the original issue: I want a simple poly fill package that falls back to a native module if it exists. The package has a single entry point, so I don’t need to (and don’t want to) expose additional paths.

The same idea applies for something like React where I might want to declaratively expose a dev and production build, depending on the env. It’s seems weird if that works for subpaths but not for the package name itself.

[ljharb]

then why specify main at all? You can have an exports with just “index.js” in it.

[jkrems]

Right, that's what this is suggesting. Maybe the confusion is that in the current implementation, exports cannot be used to specify main? We explicitly removed that feature to prevent dual-mode packages.

[ljharb]

We explicitly removed that feature to prevent dual-mode packages.

It would certainly allow packages that worked on both old and new node, but not a package that was both CJS and ESM (which is what some have expressed reservations about). I don't recall any concern about allowing a package to have different entry points for entirely different versions of node.

[jkrems]

That’s the hope! But there are some concerns that it may be confusing to
have one main for old node, one implicit main for old node (index), and one
main for new node in exports. It’s possibly confusing but I’m not sure if
we have a better option.

On Tue, Aug 6, 2019 at 12:57 PM Jordan Harband [e-mail]
wrote:

We explicitly removed that feature to prevent dual-mode packages.

It would certainly allow packages that worked on both old and new node,
but not a package that was both CJS and ESM (which is what some have
expressed reservations about). I don't recall any concern about allowing a
package to have different entry points for entirely different versions of
node.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/modules/issues/368?email_source=notifications&email_token=AAEKR5EPM4PWZI7UTGN5Z2LQDHJRVA5CNFSM4IJSAPTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WJBVQ#issuecomment-518820054,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEKR5HN6NS72L2FJSV4KM3QDHJRVANCNFSM4IJSAPTA
.

[ljharb]

If exports couldn't specify "main", but it could overwrite whatever "main" is pointing to - wouldn't that solve all of our concerns?

Dual-mode packages would still not be enabled; dual-node packages would; "main" wouldn't have to change (ie, break) its format; etc.

The only downside is that you might have to repeat the RHS of "main", in the LHS of "exports" - but that explicitness kind of seems like a pro, not a con?

[jkrems]

I think that's a reasonable outcome. Right now we have an explicit check that prevents exports from overwriting the value of main and we could just remove this check if nobody else objects.

[GeoffreyBooth]

An overwrite makes much more sense to me than an extends. In other words, if "exports" is set and defines ".", that defines the package entry point; and that’s it. "main" is irrelevant, whether or not it’s even defined. "main" would only apply to older versions of Node for such a package.

[jkrems]

@GeoffreyBooth Can you elaborate on the differences you're seeing? Is this agreement with the overall direction of this thread?

[GeoffreyBooth]

@GeoffreyBooth Can you elaborate on the differences you’re seeing? Is this agreement with the overall direction of this thread?

I was referring to the example above (simplified here):

  "main": "./path",
  "exports": {
    "./path": "./pkg.js"

The intended result in this example is what you might call a “double alias”: require('pkg') gets mapped to "./path" which gets mapped to "./pkg.js". Personally I find this confusing.

I understand that the intent of the double alias is to avoid the situation where we have two ways to define the same thing: "main" or "exports": { ".". But I think it’s simpler to just allow the two ways, and specify that "exports" takes precedence if both are declared:

  "main": "./entry-point-for-legacy-node-where-exports-is-unsupported.js",
  "exports": {
    ".": "./entry-point-for-modern-node.js"

[jkrems]

Gotcha. Yes, I think that's where the discussion ended up. So it sounds like everybody agrees. :)

[ljharb]

Almost; I’d prefer the double aliasing.

It seems we all agree on permitted use cases, just not on that one impl detail.

[GeoffreyBooth]

Yes. One thing I think we should discuss is if we _want_ “dual-across-Node-versions” packages, that is, packages that export their main entry point as CommonJS for old versions of Node and as ESM for modern versions. Such packages are currently impossible, but become no longer so via any version of specifying the main entry point through a method other than "main".

The dual package singleton hazard that @MylesBorins brought up wouldn’t apply to such packages, as by definition no single Node runtime would have two versions of the same package loaded via the same specifier. But there is at least one other hazard that I can think of, and that’s dependencies. Say there’s a CommonJS package like iced-coffeescript that depends on CommonJS coffeescript. I add "exports": { "." to coffeescript, and I don’t bump the major version because I don’t think of this as a breaking change; I’m only adding ESM support for those who want it. But suddenly iced-coffeescript, which contains code like require('coffeescript'), would be broken in modern Node because require('coffeescript') would fail—it would be routed to the ESM entry point defined in "exports": ".", and we don’t currently support require of ESM. The package would still work in legacy Node, so the author of iced-coffeescript might not even realize that their package had broken in more modern Node.

This is a story of user error, like a lot of the discussions we’ve had regarding dual packages and require of ESM, and so therefore we could ultimately decide to accept the potential for misuse and help avoid it through great documentation. But I don’t think stories like this are far-fetched or unlikely at all.

[ljharb]

I think such a user error would be surfaced quickly and fixed; i don’t think we have to worry about that.

Whether we support dual mode packages or not, I’d we don’t support dual-node packages, imo we’ve killed ESM before it starts. Without this feature, i don’t think it’s worth unflagging ESM ever.

[jkrems]

Added the agenda label. I assume the open questions are:

1. Should we support arrays in main OR should we support dot in exports?
2. If we backport CJS exports (or at least array-main) to all active LTS versions, would that change anything about our answer to (1)?

Anything else? It feels like other than the above there's general agreement on this problem and that we'd need some sort of solution.

[ljharb]

Not being an active LTS version doesn't mean it's not in use; 2 wouldn't change my answer to 1.

[guybedford]

Seems like we missed discussing this agenda item yesterday, so we should keep it around for next time.

[jkrems]

Option:

• --no-exports flag for upgrade path.
• Decide on additional property vs. accepting exports[.].

[ljharb]

Sorry i had to drop off for the last 15 minutes; can you elaborate further?

[jkrems]

@ljharb Sure, let me try to sum the two things up.

--no-exports

There's a case where with dot-main a node upgrade could fail. Scenario: One of your (potentially indirect) dependencies choose to point main to a CommonJS file and exports[.] to an ESM file. Your app works great on the pre-exports node but after the upgrade attempts to require the package fail (since exports[.] wins over main and CommonJS can't deal with the ESM file it points to). You would now be blocked in rolling out this new node version until either everything is rewritten to ESM (including potential intermediate packages) or you managed to remove the problematic dependency.

Having a simple flag to opt out of exports (e.g. --no-exports) would allow the node upgrade to be unblocked, assuming you can set it in your environment. The flag globally disables exports (just like the experimental flag today). Once your dependency tree was fixed to properly handle this scenario, you can remove the flag again.

Agreement on the call was that this would sufficiently address the issue ("we have a reasonable workaround").

Decide on additional property vs. accepting exports[.]

This referred to a potential bikeshed over where exactly dot-main would live. There have been readability concerns around exports[.] because "." on its own is somewhat light on signal. Chatting with @guybedford out-of-band one possible path would be to also reintroduce the sugar the proposal started with so that the literal . key would only appear in "advanced" scenarios where multiple paths are exported.

// equivalent:
{ "exports": "./lib/foo.js" }
{ "exports": { ".": "./lib/foo.js" }
{ "main": { ".": "./lib/foo.js", "exports": {} }

// equivalent:
{ "exports": [{ "future": "syntax" }, "./lib/foo.js"] }
{ "exports": { ".": [{ "future": "syntax" }, "./lib/foo.js"] } }

In other words: If exports is a simple string or array ("leaf value"), it's considered sugar for setting dot-main to that value.

[ljharb]

I don't consider a flag a reasonable workaround; I could be relying on exports working, and then suddenly add a dependency that forces me to use --no-exports, thereby offering no workaround except "don't use the dep", which isn't one.

[jkrems]

If you just add a dependency that uses exports and it only ships ESM but you want to require it, I'm not sure how that relates to --no-exports. That's the same problem as with a dependency that doesn't support Buffer but you need to pass it one. And its the same solution: You either can't use it or you have to change either end of the API (consumer or implementation)..?

[ljharb]

The goal is to increase compatibility, not create a forking point for the ecosystem.

[jkrems]

I don't think anybody disagrees there. Is your vote that we go back to "add array syntax to main"?

[ljharb]

No; that’s much less backwards compatible.

I think this overlaps heavily with the use case that motivates dual packages - it’s critical to be able to have a package that works on both node 10 and 14 with the same specifiers, and ideal that it can be imported and required with the same specifiers on node 14. I suspect any solution to this will address the concern, making a no-exports flag useless.

Additionally, since specifying exports is meant to prevent import/require of certain files, I’d not want a trivial node flag to be able to bypass that.

[jkrems]

I'm not sure what alternatives you see that I'm missing. From what I can tell, when introducing exports we have exactly two options:

1. "array-in-main": A package that starts using it (including the advanced features like fallbacks) cannot be compatible with environments that don't support fallbacks.
2. "dot-main": A package that starts using it (including the advanced features like fallbacks) may have a different behavior in an exports-supporting environments vs. a non-exports-supporting environment.

Any attempt I made in trying to figure out a 3rd way failed. Even when not accounting for overhead/cost. Please also note that this is about packages that explicitly made the choice to have inconsistent behavior between main and exports. We should strongly discourage that kind of design imo but I don't think there's a way to statically verify it.

[ljharb]

I think if a package has decided to be user-hostile, and we have no way to prevent it, then we have to just allow it.

Our mandate should be to make user-friendly ways easy and ergonomic, and failing that, at least possible.

[jkrems]

Is it fair to say that using the terms above you'd favor "dot-main" with "if your dependency screwed up / is user-hostile, you may need to fix it before you can upgrade node for your existing project" ("no --no-exports")?

[ljharb]

Yes, I think that's fair to say. In other words, I actively don't want a flag to bypass the dual-mode/dual-node hazard, I want to (without blocking exports) move forward looking for a solution to the wider problem of that hazard.

[MylesBorins]

I don't believe that the --no-exports flag had anything to do with the dual mode hazard. The reason for the flag was compatibility... allowing folks moving from a version of node that didn't support exports to move to a version that does support exports without having to do a wholesale refactor.

I don't think this has to do with user hostile... Folks may choose to offer legacy support via main so that they can continue to support, for example, Node.js 10 while it is still supported in LTS for another 20 months. I'm having a hard time grasping why this pattern would be problematic. The non-exports variant would be backwards compatible... it would also make sense to be a wholesale change. You are either running in the old mode or the new mode. We have lots of prior art in node of offering this type of opt-out.

[jkrems]

I don't think this has to do with user hostile... Folks may choose to offer legacy support via main so that they can continue to support, for example, Node.js 10 while it is still supported in LTS for another 20 months

The problem is when main and exports actively point to different targets, especially while not all tools (or versions of node in LTS) support exports. It means that code can surprisingly break while upgrading tools and it may be hard to fix if the issue is a package ten layers deep into the dependency tree. It doesn't even matter if its CJS and ESM. The same applies to any kind of API difference.

An example of a "good" use of main+exports could be:

{
  "main": "./lib/uuid-full.js",
  "exports": [
    "std:uuid",
    { "onlyIfWebCrypto": "./lib/uuid-small.js" },
    "./lib/uuid-full.js"
  ]
}

The major point in that example is: main is 100% the same interface and works in the same environments as something that would also be targeted via exports.

This would be the "user-hostile" version:

{
  "main": "./lib/uuid-full.js",
  // no need to care about a complete polyfill, surely environments that need one
  // would use main and not exports!
  "exports": "./lib/uuid-minimal.js"
}

Having exports resolve to ESM but main to CJS is just one example of such a package. But it's not the only one.

[jkrems]

PR: https://github.com/nodejs/node/pull/29494

[jkrems]

Closed via https://github.com/nodejs/node/pull/29494