J2cl: Understanding j2cl's building blocks - to be used with buck

[bugok]

As I mentioned in a previous question, I'm currently working on getting j2cl running with the buck build system.

I'd like to ask if my current understanding of the bazel build flow is correct, so I could imitate it with buck. I understand that j2cl is built to be integrated with bazel, but I think there should be a way to make it work, assuming that I can make standalone-CLI versions of the subcommands that bazel runs.

I'm basing my understanding on running the bazel build command on the hello world application in the repo (with the -s option). I'm using bazel version 2.0.0).

Here's the flow, as I understand it today:

```
java sources
|
| (gwt
| unsupported
| stripper)
|
source jar
|
|
| (javac)
|
| original
class jar js sources
| |
| |
| (j2cl |
| transpiler) |
| |
|----------------------------|
js sources
|
| (closure
| compiler
| and checker)
|
closured js
sources
````

Questions I have:

• How do you create the source maps - after all those transitions? Is there a way for me to imitate this?
• In j2cl_common._j2cl_transpile, I see that there are several args being passed to the transpiler. From what I see when running J2clCommandLineRunner --help, I see only the -classpath, -d and -nativesourcepath options (and not libraryinfooutput, readablelibraryinfo and others, which show up in the j2cl_common.bzl file).
• Did I misunderstand something? Is there another CLI I need which isn't listed?

Thank you!

[niloc132]

Flow chart:

• Consider running javac once before stripping, so that annotation processors are run consistently in the case of code which is shared between j2cl and javac. Probably won't matter, but it is an extra step we took in the process of building the maven plugin to make sure there were no surprises.
• The j2cl transpiler does _not_ need the bytecode of the sources which are currently being transpiled - only the gwtincompatble-stripped java sources, and the gwtincompatible-stripped bytecode of any class that those sources depend on. That is, you may need to run javac on the stripped sources, but only to give them to a later run of the j2cl transpiler on a downstream project - if you are transpiling the "most downstream" project, there is no requirement that you produce that stripped bytecode (but it could be handy for sanity checking that the code makes sense, as another way to get errors than just trusting j2cl to have good errors).
• The j2cl transpiler does not need plain .js files, only .native.js files. Other .js files must then be passed along to closure separately.

How do you create the source maps - after all those transitions? Is there a way for me to imitate this?

The J2cl transpiler will output the original .java file, any .native.js (renamed to .native_js), two .js files, and a single .js.map file describing the mappings back to those original files - no need to do anything extra. Only the .impl.java.js file will use sourcemaps, the .java.js file just worries about dependencies. The .impl.java.js file will end in a //# sourceMappingURL=WhateverClass.js.map comment to indicate where the sourcemap file is relative to itself.

In j2cl_common._j2cl_transpile, I see that there are several args being passed to the transpiler...

I can't answer this, we aren't using these at this time in maven. Check out J2clCommandLineRunner for the "full" list of options that you can use from the command line, there are two hidden ones there as well, -frontend and -readablesourcemaps.

Did I misunderstand something? Is there another CLI I need which isn't listed?

There are two other main() classes last I checked, though we aren't presently using either. Both are in tools/java/com/google/j2cl/tools:

• minifier: according to the javadocs, this should let you minify any j2cl-generated js file in such a way as to reduce file size without actually optimizing the entire application. Docs do look at least a little out of date though, since they claim that only .java.js files will be touched, which would defeat the purpose (since most of the transpiled code ends up in .java.impl.js), though the method which tests this file name is unused. It also says that you should "reuse the same minifier instance across multiple files", but the main() only accepts a single file. Source maps are not tweaked at this time, which is one of the main reasons I've ignored this so far.

* rta: The "Rapid Type Analyzer" tool, this is apparently not yet intended to be used from command line, only from within bazel as a worker. It seems to take the libraryinfooutput that you mentioned before as inputs and generate at least a report of unused files, members, lines across those libraries (or per library?). According to the BUILD file, this appears to be used inside of something labeled com/google/caribou/ui/pinto, which I'm betting is a visualization or reporting tool, based solely on context.

Some of our notes from producing the maven tooling may be helpful, here's a section in the readme that we've been sharing when discussing this topic https://github.com/Vertispan/j2clmavenplugin/blob/master/Readme.md#build-process

We have a fork of this repo with a few commits that we continuously rebase on top of it, at least one or two may be of interest to you if buck has a "worker" mechanism like bazel does, so that the transpiler can be kept running, and just be given new sources to transpile when something changes, plus to allow different naming conventions than bazel, and the output directory to not necessarily exist. In no particular order:

• https://github.com/Vertispan/j2cl/commit/c4f912e0a808c081669252c543b6c1ad32524dea
• https://github.com/Vertispan/j2cl/commit/4a4f1d8ba859860c6e4efaf27480d30c4f24e35e
• https://github.com/Vertispan/j2cl/commit/f4a832936578954ec78a8dfb954ef707935e0d49
• https://github.com/Vertispan/j2cl/commit/551b541156497d16a5844e6c6ca4fd1ab444b490
• https://github.com/Vertispan/j2cl/commit/a1c22dd68a513cc299fd080c35ecba4dc2cf583d

Again, these commits get rebased regularly, so those SHAs may change, but I think those are the only real changes to the transpiler itself that we are making for use in maven (and soon, gradle).

[rluble]

The diagram is not completely correct. For the sources being compiled, J2CL transpiles from the source jar, however to resolve dependencies it uses the class.jar produced by javac.

• How do you create the source maps - after all those transitions? Is there a way for me to imitate this?

Here is the sourcemap spec. The mapping J2CL uses for debugging is statement (sort of line) based. GwtIncompatibleStripper preserves the mapping by not removing lines. So I don't think you need to do anything special to achieve the same functionality we have now with bazel.

You could sort of mimic the structure of the bazel rules. Of course they will probably be very different since I am not sure the concept of providers is also found in buck, but the basic flow should be similar.

• In j2cl_common._j2cl_transpile, I see that there are several args being passed to the transpiler. From what I see when running J2clCommandLineRunner --help, I see only the -classpath, -d and -nativesourcepath options (and not libraryinfooutput, readablelibraryinfo and others, which show up in the j2cl_common.bzl file).
• Did I misunderstand something? Is there another CLI I need which isn't listed?

The integration with bazel supports workers and is done through BazelWorker and the arguments supported by the bazel integration are in BazelJ2clBuilder

Feel free to submit a patch to add the missing options to J2clCommandLineRunner.

W.r.t.

• Consider running javac once before stripping, so that annotation processors are run consistently in the case of code which is shared between j2cl and javac. Probably won't matter, but it is an extra step we took in the process of building the maven plugin to make sure there were no surprises.

J2cl needs the sources produced by apt. If I am not mistaken the bazel build runs the apts after stripping. And I think that is the way it should work, since annotations like GwtIncompatible are not propagated by most apts.

• The j2cl transpiler does _not_ need the bytecode of the sources which are currently being transpiled - only the gwtincompatble-stripped java sources, and the gwtincompatible-stripped bytecode of any class that those sources depend on.

Correct. The bytecode is only to resolve references and of course it needs to be the one after stripping.

• The j2cl transpiler does not need plain .js files, only .native.js files. Other .js files must then be passed along to closure separately.

Correct.

[bugok]

@niloc132, @rluble: Thank you both very much for taking the time to write such detailed replies! This is definitely making my life easier.

[niloc132]

Feel free to submit a patch to add the missing options to J2clCommandLineRunner.

Revisiting an old topic here, but I don't think I got a straight answer before: would you accept patches which made it possible to do what the j2cl bazel worker is doing? Namely, making types/members public so that maven/gradle/buck/etc could run a jvm and ask it to recompile over and over, instead of using the J2clTranspiler main?

J2cl needs the sources produced by apt. If I am not mistaken the bazel build runs the apts after stripping. And I think that is the way it should work, since annotations like GwtIncompatible are not propagated by most apts.

The issue in maven is that jars are published already containing their generated sources - we don't re-run processors when j2cl'ing them. Either it would be required that maven (and ... gradle) always publish their annotation processor dependencies in a way which is consistent with recompiling the dependency, and also avoid putting generated sources in their sources jars, or projects would have to be fine that sometimes we run APT before gwtincompatible-strip, and sometimes after (depending on whether or not it is in your reactor or not).

[rluble]

Revisiting an old topic here, but I don't think I got a straight answer before: would you accept patches which made it possible to do what the j2cl bazel worker is doing? Namely, making types/members public so that maven/gradle/buck/etc could run a jvm and ask it to recompile over and over, instead of using the J2clTranspiler main?

You mean adding another entry class, e.g. MavenJ2clBuilder? I don't see an immediate reason to object to that. But in general we would not wan't to arbitrarily extend the contract for J2CLTranspiler, using the existing contract, on the other hand, should be acceptable.

[niloc132]

Right, I don't think adding new classes to the upstream google/j2cl repo for each potential downstream build tool makes sense, and would limit build tools that didn't get the ability to add their own classes, dependencies as needed. Instead I'm hoping to take the J2clTranspiler.transpile(options) and let it be called rather than introducing a proxy for it:
a1) make J2clTranspiler public,
a2) make its transpile(options) public,
b) add more factory methods for FileInfo (the only existing one assumes files come from jars).

Those at least get us to the point where we can build, plus or minus c) having to rename jar files to read their contents.

The other two patches are nice-to-have: d) allowing the J2clTranspilerOptions object to be made into a builder (easy to copy rather than create from scratch each time), and e) ensuring that a directory exists before attempting to write to it.

[bugok]

@rluble:

J2cl needs the sources produced by apt

What's 'apt' in this context?

[niloc132]

APT refers to java Annotation Processors, introduced in about Java 1.5, allowing code to be generated at compile time.

[gkdn]

@niloc132 Any reasons to not use the contract of main which is already public?

[niloc132]

The System.exit at the end of CommandLineTool.execute() makes it difficult to use J2clCommandLineRunner.main() from a bazel-worker style long-running jvm - the idea is that we want to get the status code and problems back. Ideally using the options builder can be nice too, since it gives more control over the use of zips (where it makes sense) or the names used in the paths for errors (when files get copied to temp dirs, etc), though I do understand wanting to hide those implementation details.

(edit: removed sarcasm, it doesn't always come across as funny-sounding as it seemed in my head)

[niloc132]

If something like CommandLineTool.processRequest were made public, that would nearly do the job, with the exception of something like Problems then being part of the public API - some way to encode the set of info, warning, error messages and possibly the status code (though today at least that is just checking if the set of error messages is non-empty). Not my favorite solution as it does limit our access to controlling FileInfo and getting slightly nicer feedback that way, but it could work.

[gkdn]

I think we can provide an error code returning method next to main which should be easy to maintain without concerns about breaking any more than the command line interface.

[tbroyer]

For reference, those "error code returning method next to main" have been added in 0c6a88cbb896e3c0030b5d51af205a211eb089e8 (approx a month ago) for GwtIncompatibleStripperCommandLineRunner (with additional fix in 5606d349d0c05d44a4e99bd44090531f91e036cb) and J2clCommandLineRunner.

I also wrote about J2CL's building blocks and how they're used with Bazel in https://blog.ltgt.net/reverse-engineering-j2cl-bazel-integration/ (doesn't include RTA and Minifier as they're not used in the "Open Source J2CL")

[niloc132]

RTA and Minifier are still packaged and pushed to maven however, and at least minifier looks like it can be fit into the pipeline with only minimal other changes, but the work hasn't happened yet (plus it actually has a command line tool, though the args are missing the magical j2cl_rta_removal_code_info_file/j2cl_rta_pruning_manifest system property, without which you won't get deterministic renaming...). The current BUNDLE-per-dependency approach doesn't appear it would make a ton of sense for RTA, but there are other lines of inquiry where it may work for us...