Buck: Support for Java 11

[shs96c]

Buck should work with the latest LTS Java release. It does not. Java 8 is being EOLd in January, so waiting much longer for support isn't really an option.

[shs96c]

Currently, buck fails like so:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.Error: java.lang.ClassNotFoundException: com.facebook.buck.cli.bootstrapper.filesystem.BuckFileSystemProvider
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:141)
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:111)
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:109)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:109)
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>(FileSystems.java:103)
    at java.base/java.nio.file.FileSystems.getDefault(FileSystems.java:190)
    at java.base/java.io.File.toPath(File.java:2290)
    at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1222)
    at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:726)
    at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:843)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:246)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:176)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:346)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:804)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:749)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:742)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:741)
    at java.base/jdk.internal.loader.URLClassPath$JarLoader.<init>(URLClassPath.java:716)
    at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:484)
    at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:467)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:466)
    at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:435)
    at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:304)
    at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:696)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:622)
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:580)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:760)
    at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:655)
Caused by: java.lang.ClassNotFoundException: com.facebook.buck.cli.bootstrapper.filesystem.BuckFileSystemProvider
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at java.base/java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:131)
    ... 34 more

[jtorkkola]

We're actively working on migrating to Java 11 by EOY. There are various issues to solve.

[brettwooldridge]

Is there any workaround for this? For example, running buck under Java 8 but compiling under Java 11?

[brettwooldridge]

@jtorkkola What are the blocking items here?

[brettwooldridge]

@jtorkkola Bump. I am willing to take a look at some of the Java 11 issues, but it would be useful to know what the known items are.

[jtorkkola]

@brettwooldridge Sorry for the delayed response (was on Thanksgiving holiday). I'd love to get some help.

The status is that buck can be built against Java 10 with ant (point JAVA_HOME at a Java 10 installation, and run ant clean default), and that produced build of Buck can be used to build non-Java code pretty well (you just need to pass the --java10-test-mode flag to buck). We started with Java 10, as Java 11 wasn't available when we started, and we generally expected the move from Java 10 to Java 11 to be relatively easy. With building Java code, there are primarily issues with in-proc compilation (custom compiler JARs and ABI generation). I'm concentrating on this. There are also miscellaneous minor issues.

As for Java 11, there is currently a mysterious annotation processor issue when building Buck. If you point JAVA_HOME at Java 11 and run ant clean default, you'll see lots of missing symbol issues for classes we expect to be generated by Immutables. I looked into this a bit, and it looks like sometimes Immutables sources do get generated, but they don't get picked up by the compiler for some reason, and sometimes they just don't get generated (tried with both ECJ and javac; the behavior varies a bit). It's possible ECJ doesn't properly support Java 11 annotation processing yet for one thing. I could definitely use someone's help to investigate further.

[brettwooldridge]

@jtorkkola Thanks for the info. Yeah, I had already run into the annotation issue when trying to compile (source files had been generated, but not compiled), and was investigating whether the newest version of ECJ fixes the issue. I was curious about switching back to javac, but your reply tells me there are still issues even so.

But if ECJ/javac is the main blocking issue, I feel like I can make some progress by diving deeper.

[brettwooldridge]

@jtorkkola

Status update:

Still pulling the thread ... it looks like an issue/bug in the Immutables library. Adding the -XprintProcessorInfo and -XprintRounds options to the ecj compiler is now showing this error generated by the immutables library:

ERROR: Generated source file name colission. Attempt to overwrite already generated file: com.facebook.buck.query.TargetLiteral, javax.annotation.processing.FilerException: Source file already exists : com.facebook.buck.query.TargetLiteral. If this happens when using @Value.Immutable on same-named nested classes in the same package, use can use @Value.Enclosing annotation to provide some sort of namespacing

This error occurs for roughly 100+ classes, one of them being:

ERROR: Generated source file name colission. ... com.facebook.buck.step.ExecutionContext, javax.annotation.processing.FilerException: Source file already exists : com.facebook.buck.step.ExecutionContext. If this happens when using @Value.Immutable on same-named nested classes in the same package, use can use @Value.Enclosing annotation to provide some sort of namespacing

Where ExecutionContext just happens to be the first error reported by the compiler:

[javac] ----------
[javac] 1. ERROR in /Users/brettw/Documents/dev/buck/src/com/facebook/buck/android/AabBuilderStep.java (at line 27)
[javac]     import com.facebook.buck.step.ExecutionContext;
[javac]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[javac] The import com.facebook.buck.step.ExecutionContext cannot be resolved

So, I am now going to debug into the immutables library. Luckily, the misspelling of "colission" in the error message makes it relatively easy to find where the error is being generated as a starting point.

[brettwooldridge]

@jtorkkola Tracking bug on immutables is here.

[jtorkkola]

@brettwooldridge That's great. Thanks for looking into this!

[kageiit]

@jtorkkola are the abi issues specific to source-only abi mode?

[jtorkkola]

@kageiit Both source ABI and source-only ABI are the problem areas, yes.

[jtorkkola]

@brettwooldridge Upgrading to ECJ 4.10 fixed the Immutables issue. Thanks for your help!

After submitting a few other fixes, Buck now builds on Java 11, and can be run with the --java11-test-mode flag. Please note that this mode is experimental and unsupported, and you need to run buck clean before and after using it. But if you try it out, please do report any issues.

In-process Java compilation and source ABI generation are in much better shape now, but I expect to find a few more issues with additional testing.

[brettwooldridge]

The latest buck build works for me. That is to say, buck itself is built with JDK 8, but at runtime JDK 11 is used to build our project.

[jtorkkola]

--java11-test-mode has been removed and no longer needs to be specified when running Java 11 builds of Buck.

[v-jizhang]

Java 11 supported.