Bazel: Bazel build crash with IllegalArgumentException

[ulfjack]

This is bazel 0.22.0, on an otherwise clean machine.

$ bazel build //java/info/adams/self
Starting local Bazel server and connecting to it...
INFO: Invocation ID: d4aa1a5e-eba4-4f08-b033-f974134d23b2
INFO: Analysed target //java/info/adams/self:self (16 packages loaded, 389 targets configured).
INFO: Found 1 target...
ERROR: /home/ulfjack/.cache/bazel/_bazel_ulfjack/d8cef0f170ae4455e05403d45c33884b/external/bazel_tools/tools/jdk/BUILD:188:1: SkylarkAction external/bazel_tools/tools/jdk/platformclasspath.jar failed (Exit 1) java failed: error executing command external/remotejdk_linux/bin/java -XX:+IgnoreUnrecognizedVMOptions '--add-exports=jdk.compiler/com.sun.tools.javac.platform=ALL-UNNAMED' -cp ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Exception in thread "main" java.lang.IllegalArgumentException: external/local_jdk
        at jdk.compiler/com.sun.tools.javac.file.Locations$SystemModulesLocationHandler.update(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.file.Locations$SystemModulesLocationHandler.handleOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.file.Locations.handleOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.file.BaseFileManager.handleOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.file.BaseFileManager$2.handleFileManagerOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.main.Option.process(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.main.Option.handleOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.file.BaseFileManager.handleOption(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.main.Arguments.doProcessArgs(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.main.Arguments.processArgs(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.main.Arguments.init(Unknown Source)
        at jdk.compiler/com.sun.tools.javac.api.JavacTool.getTask(Unknown Source)
        at DumpPlatformClassPath.dumpJDK9AndNewerBootClassPath(DumpPlatformClassPath.java:106)
        at DumpPlatformClassPath.main(DumpPlatformClassPath.java:67)
Target //java/info/adams/self:self failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 36.836s, Critical Path: 3.83s
INFO: 1 process: 1 linux-sandbox.
FAILED: Build did NOT complete successfully

[ulfjack]

This seems to be Debian 9.6 (it's a cloud vm, not my personal machine).

[dslomov]

@lberki could you take a look/assign to someone who can take a look?

[meisterT]

I guess this is the same as #6993 - is there a JDK installed on the machine? We definitely need to improve the error detection there and not crash bazel.

[ulfjack]

Yes, there is:

$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

[ulfjack]

It looks to me like the Java rules are generating an incorrect command line for the DumpPlatformClassPath tool, but it might also be missing error handling there?

[meisterT]

Can you please try with --java_toolchain=@bazel_tools//tools/jdk:toolchain_java8?

[ulfjack]

Same exception.

[philwo]

DumpPlatformClassPath.dumpJDK9AndNewerBootClassPath(DumpPlatformClassPath.java:106)
vs.
openjdk version "1.8.0_181"

🤔

[lberki]

/cc @cushon (the earliest I can take a look is tomorrow -- it's very late here already and I'm somewhat exhausted from a day of conferencing)

[lberki]

Well, that's not good. I tried this on a debian:stretch image and I got a _different_ exception:

Internal error thrown during build. Printing stack trace: java.lang.IllegalStateException: //java/a:a BuildConfigurationValue.Key[c4592abc3e3
009cd244247c661481fda] false -> ErrorInfo{exception=com.google.devtools.build.lib.skyframe.FileSymlinkInfiniteExpansionException: Infinite sy
mlink expansion, rootCauses={FILE:[/root/.cache/bazel/_bazel_root/887904812217cca9bc2b9adb875daf42/external/local_jdk]/[.cache]}, cycles=[], 
isCatastrophic=false, rootCauseOfException=FILE:[/root/.cache/bazel/_bazel_root/887904812217cca9bc2b9adb875daf42/external/local_jdk]/[.cache]
, isDirectlyTransient=false, isTransitivelyTransient=false}
        at com.google.common.base.Preconditions.checkState(Preconditions.java:823)
        at com.google.devtools.build.lib.skyframe.SkyframeBuildView.assertSaneAnalysisError(SkyframeBuildView.java:660)
        at com.google.devtools.build.lib.skyframe.SkyframeBuildView.processErrors(SkyframeBuildView.java:525)
        at com.google.devtools.build.lib.skyframe.SkyframeBuildView.configureTargets(SkyframeBuildView.java:444)
        at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:369)
        at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.runAnalysisPhase(AnalysisPhaseRunner.java:209)
        at com.google.devtools.build.lib.buildtool.AnalysisPhaseRunner.execute(AnalysisPhaseRunner.java:118)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:143)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:253)
        at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:83)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:477)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:205)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:749)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$1600(GrpcServerImpl.java:103)
        at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:818)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

[lberki]

Uh. So turns out, the crash that happens is different when there is no java, _but_ the WORKSPACE is in $HOME. Putting WORKSPACE in a subdirectory reproduces the previous error.

[lberki]

@lfpino : can you take a look?

This appears to happen because DumpPlatformClassPath doesn't like when @local_jdk doesn't actually point to a JDK, for example, when there is no JDK installed locally.

One possible fix is that before DumpPlatformClassPath does anything substantial, it checks whether it actually has a JDK and if not, raises and error. After all, Bazel can't be expected to build Java binaries targeting a nonexistent JDK...

Then, once the fix is there, it's probably patch release time :(

/cc @aehlig

[cushon]

DumpPlatformClassPath.dumpJDK9AndNewerBootClassPath(DumpPlatformClassPath.java:106)
vs.
openjdk version "1.8.0_181"

I think that part is actually OK, the JDK9 in dumpJDK9AndNewerBootClassPath refers to the host JDK, which should is the remote JDK 9.

There's some related discussion of improving diagnostics for missing/incorrect java runtime configuration in #6118,

[lfpino]

Investigating.

[philwo]

Tobi and me found the problem, a fix and a workaround. :)

The problem: When JAVA_HOME is set to point to a JRE 8 instead of a JDK, Bazel will happily try to use it, but then chaos unfolds in DumpPlatformClassPath:

1) This method tries to detect whether the target (= system) JDK is a) Java 8 or b) Java >=9: https://source.bazel.build/bazel/+/28841a6c4fe303e661041e479f78ed8197a82548:tools/jdk/DumpPlatformClassPath.java;l=95

2) It does so by making the assumption: If we look for well-known files of OpenJDK 8 and they don't exist, then it must be an OpenJDK >= 9. This check obviously also comes to the conclusion that you're looking at OpenJDK >= 9 when you're not looking at a JDK at all (like a JRE).

3) DumpPlatformClassPath passes the path to its logic for OpenJDK 9 boot classpath detection, which then crashes with an InvalidArgumentException, because it refuses to work with something that's not an OpenJDK >= 9.

The fix: https://github.com/bazelbuild/bazel/pull/7324

The workaround: Set JAVA_HOME to a JDK on platforms where that's not the case, like in the Cloud Shell container. I verified that on Cloud Shell the JAVA_HOME is set incorrectly to a JRE:

philwo@cloudshell:~ (bazel-public)$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64/jre

By setting it to the JDK (/usr/lib/jvm/java-8-openjdk-amd64) instead, Bazel works fine.

[philwo]

I also sent feedback to the Cloud Shell team so that they are aware of this and can fix the JAVA_HOME in their container. The current situation will probably also cause other build tools like Maven to fail, so fixing this is a good idea anyway. (Internal reference: b/123752456).

[ulfjack]

Yep, that works! Thanks!

[gordinmitya]

In my case it turns to be missing the jdk at all, so

sudo apt install default-jdk

solved the problem