Closure-compiler: Add support for "closure bundled" and per file transpiled ES6 modules

[jplaisted]

We've been discussing ES6 module support. Here is the current design for library and compiler work. The compiler has partial support for ES6 modules currently, however it needs to be reworked for bundling and to reflect changes to the Closure library.

tl;dr

• goog.require will support relative paths for goog and ES6 modules from goog modules and ES6 modules (though ES6 -> ES6 should use import not goog.require).
• ES6 modules will need to be transpiled to goog modules when whitespace bundling. Currently whitespace bundling is not supported for ES6 modules.
• The above two bullets combine so that when whitespace bundling mixed Closure / ES6 module code only ES6 modules need to be _lightly_ transpiled to goog modules. Imported paths will become goog.require'd paths.
• There will be a new goog.bootstrap function as a replacement for "bootstrap goog.requires" (top level gog.requires when loading code via the debug loader). Because loading will be asynchronous there needs to be an alternative to goog.require that provides a callback when things have loaded.

[jplaisted]

Library tracking issue is here. https://github.com/google/closure-library/issues/846

[ChadKillingsworth]

ES6 modules will need to be transpiled to goog modules when whitespace bundling. Currently whitespace bundling is not supported for ES6 modules.

This probably needs restricted to internal or controlled by some sort of flag. We don't want ES6 module transpilation externally to have a hard dependency on Closure-library - even in Whitespace Only mode.

[jplaisted]

Ah, good point. I think having one pass that transpiles ES6 modules to goog.modules is still a good idea to ensure that there is only one code path to transform ES6 modules in the compiler. If your input is only ES6 modules then we could just run an additional closure_pass over the transpiled goog.modules to get rid of the goog.* calls. It's a little more cycles to do this in two passes but cuts down on code that we need maintain.

We'll just have to be sure to do this in strip_whitespace_and_comments_only and output_bundle.

[ChadKillingsworth]

There is a great amount of pre-existing code that depends on modules being transpiled/bundled as global scripts - even in WHITESPACE_ONLY mode. How is this going to be effected? For instance, I use WHITESPACE_ONLY mode with the preserve type annotations option to bundle modules for input into Polymer's analyzer (it doesn't understand modules).

If I understand the goal here, the idea would be to produce some type of bundle that is in itself loadable as a module. I've given a lot of thought to that topic. We need to be able to specify the type of output module. For instance, I should be able to input mixed modules, bundle them, then output them in the module format of my choice: ESM, CommonJS, goog.module or global script.

[jplaisted]

Again, if you want global code then run closure pass. We'll just have to be sure to provide options for closure pass with strip_whitespace_and_comments_only + output_bundle. These will default to outputting goog.loadModule calls.

If I understand the goal here, the idea would be to produce some type of bundle that is in itself loadable as a module.

Nope, not the current goal. What would exports be? How do you reference the bundle by path (it only has one path are are you meant to be able to still reference the "files" contained in the bundle?)? Current goal is just to get a bunch of ES6 modules into one bundle that would be considered a "binary" rather than a "library" (self contained code that isn't meant to be imported elsewhere and can run as a script, not a module. e.g. and entire web app).

Polymer's analyzer may not understand modules but at the same time no one is going to be able to use your bundle if they're required to reference compiled code.

Maybe that's a longer term goal, but this is get base ES6 module support. So ES6 Modules and/or Closure Modules in => Valid script out.

[ChadKillingsworth]

Nope, not the current goal. What would exports be?

I have plans for defining just that. Haven't written them down yet though.

[rictic]

We will have support for ES6 modules in polymer-analyzer. Here's our tracking bug: https://github.com/Polymer/polymer-analyzer/issues/731

[thheller]

If I could make one suggestion it would be to not change the behaviour of goog.require but instead introduce something new like goog.requireFile.

I'm not sure how many internals that involves but bolting an entirely new concept onto an old one doesn't seem ideal to me.

One a side note: That is pretty much what I did recently.

My solution had slightly different goals but I have full support for using npm dependencies in CLJS code. Since the CLJS compiler emits Closure JS style code that by extension includes the Closure Library.

I don't want to go into too much detail here but I do think there is some overlap here. I will definitely follow along to see where you take this.

[jplaisted]

Since we're currently only allowing relative paths it isn't that difficult (see if argument begins with ./ or ../). Presumably even if we allowed relative paths it wouldn't be that bad either (see if argument contains /, though this currently isn't illegal, which is an issue).

Keep in mind this still acts like old goog.require. It isn't meant to import just any file, so you cannot use it to import some JSON object definition file. It's only meant to require other Closure files or, in the near future, ES6 modules.

The idea was to attempt to have consistency with dependency management through Closure. Whether it is by path or by namespace it functions the same; it will retrieve the exports of the module and return it.

Also we are adding a new mechanism for bootstraping closure code in uncompiled environments. Previously you could use a good.require to bootstrap your application and it would be available to the next immediate script tag. This is no longer the case with ES6 modules so we are adding a goog.bootstrap that accepts a callback to call once things have loaded. This is enough of a behavioral change to justify a new call. Plus we can ban this call in compiled code :sunglasses:

Also while I'm here I should mention the first commit has gotten in. Just linking this here so people can stay up to date on the commits, not to stifle discussion. If we feel we need some goog.requireFile we can switch over to that instead.
commit c44d17cfb3e06409cb9d32dd5f5fe6bc9cafced4

[jplaisted]

So upon further thought yes it's a good idea that transpiled ES6 modules do not have a dependency on Closure library. So we'll need to transform them to some new $jscomp.module that is similar to goog.module and inject a runtime to handle loading of modules. Then base.js will have to interop between ES6, $jscomp, and goog modules correctly.

I've update the document on this.

[jplaisted]

So I've still be flopping around a lot what ES6 modules should be transformed to as an intermediary.

At first I said goog.modules (first post), which were ruled out to eliminate a dependency on Closure Library.
Then I said some new $jscomp.module (post above).

There's still been some discussion around CommonJS and I'm not entirely convinced.

Chad I guess I'm still not clear on what your main point is, but I'm trying to understand. :smile: Can you clarify? Is it to use CommonJS as an intermediary or to eventually have the compiler output CommonJS/ESM/whatever (in which case we need not just ES6 module => CommonJS transformers but [Any valid input] => [Any valid output] transformers, which is a lot, and probably not worth blocking Closure Library / whitespace bundled ES6 modules over).

General dump of thoughts:

• “We should transform everything to a common thing internally and then from there transform to globals [or whatever output we want]”
  
  
  
  • Pros:
  
  
  • Changes to the final step benefit everything
  
  
  • Bundled output / minimally transformed output is consistent no matter the input
  
  
  • Can interop anything / anything
  
  
  • Cons:
  
  
  • Can interop anything / anything (may not be wanted, eg goog.provide -> ES6), unless this is enforced before the common transformation
  
  
  • Still need A -> Common and B -> Common and C -> Common transformers. Adds unneeded indirection in the compiler.
  
  
  • Issues if there isn’t anything that couldn’t be transformed (not sure that there is atm). Exceptions start to defeat the purpose of this common functionality.
  
  
  • Need a TON of testing to ensure we don't break existing cases. Would be a very slow going change. Whereas if we just change how ES6 modules are handled we only need worry about that case (and its interops).
  
  
• “Outputting CommonJS for ES6 modules in whitespace only (or something close to it) allows users to take advantage of this output”
  
  
  
  • The compiler isn’t a generic JS transformer. It is mean to spit out optimized JS. Not generic A:B transformation. Go use some other tool (babel?). Whitespace only should only be used for readability, not to rely on any side effects as to how things were made to be concatenated.
  
  
  • However this would allow a future where we can spit out optimized CommonJS modules.
  
  
  • But what does this even look like? 1:1 mapping of files as output? Or a giant CommonJS module with things exported?
  
  
  • Why not just do a full compile with compiler exports and write your own CommonJS wrappers?
  
  
  • Even if the above isn’t enough of a reason to not use CommonJS if we do do this it encourages people taking advantage of it, which could be an issue down the road. “Different purposes should have different implementations for the sake of being antagonistic”.
  
  
• “Outputting CommonJS for ES6 modules allows nice interop with real CommonJS modules”
  
  
  
  • Yeah but it means the interop between ES6 and goog provide/modules is whatever the existing CommonJS interop with goog provides/modules is, and I have no idea what that is (does anyone?).
  
  
  • Jscomp modules need interop with goog modules, provides, and CommonJS (so that ES6 has this interop). Need to write it in a few places. Might lead to a chicken and egg situation if you have all 3?
  
  
• “CommonJS modules have nice dynamic imports. We need dynamic imports eventually.”

[ChadKillingsworth]

Chad I guess I'm still not clear on what your main point is, but I'm trying to understand. 😄 Can you clarify?

My point was that everyone besides Google uses CommonJS. That's what ES modules are transpiled to for use in a browser in every open source tool I know. I am very leery of inventing a 4th representation.

What to transform things into depends greatly on what you are doing. Per-file transpilation is completely different than a whole-world optimizing bundle. I rarely use the compiler for per-file transpilation.

Need a TON of testing to ensure we don't break existing cases. Would be a very slow going change. Whereas if we just change how ES6 modules are handled we only need worry about that case (and its interops).

Changes you make to ES6 module representation are quite likely to break the interop with CommonJS that I (and many others) already depend on. That interop got broken in July inadvertently and I've been playing damage control ever since. We already have interop - Google just doesn't use it.

The compiler isn’t a generic JS transformer. It is mean to spit out optimized JS. Not generic A:B transformation.

That's exactly what transpilation is/does though. It's primary use cases externally are dependency ordering and module rewriting.

But what does this even look like? 1:1 mapping of files as output? Or a giant CommonJS module with things exported?

In per-file transpilation, 1-1. In a full compile, all modules in the same output chunk are transformed into a single module. I have this working with webpack today.

Why not just do a full compile with compiler exports and write your own CommonJS wrappers?

We don't have any methodology in the compiler today for outputting proper modules of any format (except in whitespace mode). I'm working towards that goal, but it's still a ways out. This is currently very hard to accomplish.

if we do do this it encourages people taking advantage of it.

So I hear an opposite complaint from Googlers all the time. Namely "why does Google have to do everything different and make things harder?"

CommonJS interop with goog provides/modules is, and I have no idea what that is (does anyone?).

The interop between ES modules and CommonJS is well known - and I have a PR queue'd up to improve it. I can go over that easily. The interop between CommonJS and goog.module I'm not sure - because I've never tested it. goog.module is really only used at Google and I've steered my team very clear of it. However I've never seen anything that indicates that interop would be a problem - I expect it already just works.

It is a bit unclear of what to do for whitespace only or minimally bundled modes. All other tools I know of though transpile ES6 to CommonJS for use in a browser. Other module formats should be left as-is here.

Might lead to a chicken and egg situation if you have all 3?

I don't see how. You are only allowed to specify 1 type of export. We recognize the primary module type in a very directed way - I transition between module types regularly without problems.

[jplaisted]

Thanks for the response! I wasn't thinking of chunked output, and that's a good use case for having non-script output (e.g. CommonJS, ES6 module). I think I now see some merits for having CommonJS whitespace modules as well (they'll have to be wrapped, but the injected require can fall back to a native require, if there is one, which is nice).

I think CommonJS is important and I'm in no way talking about dropping any support for it. If we break it we rollback the breaking change. Ideally we never break it. But internally Google doesn't use CommonJS at all with the compiler (afaik). goog modules and provides are thus used and used widely internally. So is per file transpilation and whitespace bundles (WHITESPACE_ONLY output and output from ClosureBundler). So we have to support all these cases if we ever want to move Closure Library (and any other common libraries within Google) to ES6 modules.

However I think I see what you're saying with the ideal end goal of the compiler. It'd be nice to have chunked output that are CommonJS or ES6 modules or whatever else we'd like. It'd even be nice to have a whitespace bundle that has wrapped CommonJS modules so that you can require standard modules that weren't compiled (e.g. require('fs')). However this is totally out of scope for what this issue is tracking. I'm renaming this issue, and Chad it would be great if you opened a new one to track that (e.g. 'Support [ES6, CommonJS, etc] module output in compiled and whitespace modes'). This involves moving goog.provide/module, ES6 module, and CommonJS module rewriting to the last step in the compiler; and rewriting them to whatever the requested output is.

We don't have the resources to block using ES6 modules internally at Google on this lofty goal. I think its worth working towards it and getting there one day. But we need something for ES6 modules sooner rather than later, even if it does get rewritten / thrown out in the future.

As a reminder the root issues here:

• The compiler doesn't understand modules (ES6, Closure, CommonJs, etc). They're rewritten early and everything is performed on globals. So ES6 needs to be rewritten to something early so that the compiler understands it.
• Google internally uses whitespace only (minimal AST manipulation with checks), closure bundler (no AST manipulation if possible, no checks), debug loader (uncompiled, unbundled, everything loaded by base.js; per file transpilation where needed), and full compiled modes. All of these have to be supported. Imagine project B depends on library A. If A wants to use ES6 modules but B is old and using goog files and the debug loader then A cannot upgrade unless B's use cases are supported or A breaks B.
  
  
  
  • Supporting whitespace only and closure bundler with ES6 modules involves rewriting them to something that can interop with goog.proviedes and goog.modules at aren't rewritten.
  
  

So with that being said ES6 modules need to get rewritten to something early that the rest of the compiler can understand AND whatever this will probably get output in whitespace only modes (for now). And in whitespace only it will need to be wrapped in something.

I had proposed originally goog.modules and this was turned down since there's a dependency on Closure Library. I then proposed some $jscomp.module that worked a bit like goog.module. I think now there's three options:

• Have a $jscomp.module that isn't similar to CommonJS to fit only the requirements we currently need.
• Have a $jscomp.module that is very, very similar to CommonJS. Literally CommonJS in all but name. This is to make moving to CommonJS in the future easier, but at the same time it also hopefully discourages people taking advantage of it in whitespace mode.
• Just use CommonJS with a wrapper.

To get back to this point:

So I hear an opposite complaint from Googlers all the time. Namely "why does Google have to do everything different and make things harder?"

The idea is "you asked for _whitespace_, not for _CommonJS_", so we shouldn't give you CommonJS, or people will use it and complain if/when we take it away. However since this seems like the direction we want to go in (where you can ask for the type of modules in a whitespace bundle, or just have things wrapped but ultimately untransformed) perhaps it's moot.

I'm inclined to the latter two options above now. Second option nicely discourages people from using it, and can easily be moved to CommonJS in the future. The downside is you won't be able to require any standard libraries that aren't compiled, though I'm not sure you can do that today (e.g. import fs from 'fs'). Using literally CommonJS has an issue that require('commonjs') and require('es6') have different semantics that we'll have to handle.

tl;dr - CommonJS / ES6 module output is a great goal to work towards but we cannot block the use of ES6 modules internally on it. Google uses goog module / provides and different compile modes internally than what is used externally (CommonJS) and we need to support everything.

Chad - I'd love if you signed off on this to make sure we're on the same page now.

[jplaisted]

Actually giving even more thought to this:

• WHITESPACE_ONLY currently has working interop with CommonJS and ES6 modules (note: none with Closure/ES6 modules).
• ClosureBundler has zero support for ES6 modules.
• The in browser transpiler has zero support for ES6 modules.

Fundamentally those last two are doing the same thing; transpiling files in isolation. And they both lack support for ES6 modules. So why not just change them and leave WHITESPACE_ONLY totally alone (except add support for Closure/ES6 module interop with path based goog.require)?

That sounds like the best approach since internally I doubt WHITESPACE_ONLY is used widely and even if it is it will still work as it does today. But since it is probably more widely used externally there's less of a change of breaking it.

And the latter two are used internally and are what need fixing. So let's only fix what is broken (or rather add features to what is missing them).

We can still use a similar approach to above with a $jscomp module that acts like CommonJS that Closure's base.js can understand. But it won't show up in WHITESPACE_ONLY.

[ChadKillingsworth]

I'm on board with your latest set of suggestions 👍 If we can do this, then the next step would be to get type checking for this type of module to the point that we can move most module rewriting into the optimizations.

[jplaisted]

I updated the document to reflect these changes. I also added a work checklist there to make it easier to track status.

[jplaisted]

Should be now be supported with fa7cb573c14912e1cb68dbbb5502330c3eeac0cf.

[jplaisted]

Actually, there's still an issue that the output_bundle of the compiler is not correct. Looks like the module runtime is missing.