Bazel: Bazel crash when run from IntelliJ plugin

[ulfjack]

When I try to sync from the IntelliJ plugin, I get this error:

Syncing project...
Updating VCS...
Running Bazel info...
Command: /home/ulfjack/Software/bazel info --tool_tag=ijwb:IDEA:community --curses=no --color=no --noexperimental_ui --noprogress_in_terminal_title --



Server terminated abruptly (error code: 14, error message: '', log file: '/home/ulfjack/.cache/bazel/_bazel_ulfjack/2cd6dd7b44f5e35bddec3984f7fcd2e4/server/jvm.out')

Sync failed

==== TIMING REPORT ====

Sync: 387ms
    BazelInfo: 367ms

[ulfjack]

The jvm.log contains:

java.lang.StringIndexOutOfBoundsException: String index out of range: -14
        at java.lang.String.substring(String.java:1967)
        at com.google.devtools.build.lib.runtime.BlazeRuntime.getRequestLogString(BlazeRuntime.java:605)
        at com.google.devtools.build.lib.runtime.CommandExecutor.exec(CommandExecutor.java:50)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:863)
        at com.google.devtools.build.lib.server.GrpcServerImpl.access$2100(GrpcServerImpl.java:106)
        at com.google.devtools.build.lib.server.GrpcServerImpl$4$1.run(GrpcServerImpl.java:929)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

[ulfjack]

It also happens at head. It's not obvious from the code what might be wrong. In particular, I don't see how either of the variables could be -14, though it can be negative if the client passes "--client_env".

[ulfjack]

The string is "--client_env=^E". Hexdump:

00000000  6a 61 76 61 2e 6c 61 6e  67 2e 52 75 6e 74 69 6d  |java.lang.Runtim|
00000010  65 45 78 63 65 70 74 69  6f 6e 3a 20 2d 2d 63 6c  |eException: --cl|
00000020  69 65 6e 74 5f 65 6e 76  3d 05 0a 09 61 74 20 63  |ient_env=...at c|
00000030  6f 6d 2e 67 6f 6f 67 6c  65 2e 64 65 76 74 6f 6f  |om.google.devtoo|
00000040  6c 73 2e 62 75 69 6c 64  2e 6c 69 62 2e 72 75 6e  |ls.build.lib.run|
00000050  74 69 6d 65 2e 42 6c 61  7a 65 52 75 6e 74 69 6d  |time.BlazeRuntim|
00000060  65 2e 67 65 74 52 65 71  75 65 73 74 4c 6f 67 53  |e.getRequestLogS|
00000070  74 72 69 6e 67 28 42 6c  61 7a 65 52 75 6e 74 69  |tring(BlazeRunti|
00000080  6d 65 2e 6a 61 76 61 3a  36 30 36 29 0a 09 61 74  |me.java:606)..at|

[ulfjack]

That's ascii code 5.

[ulfjack]

I'm not sure where that broken environment comes from, but we should safeguard against that, and not crash.

[ulfjack]

I have a fix for the crash.

[ulfjack]

Note that this change doesn't make it work, as Bazel still returns an error about the environment being broken. I'm not sure if we should change that or not.

[harmank]

@ulfjack Any update on the issue? I am facing the exact same problem.

[ulfjack]

No update on the Bazel side. I have a small patch that'd technically make it work, but maybe we should file a bug against the intellij plugin instead, since it's passing a weird environment to Bazel.

[philwo]

@brendandouglas Any idea why the plug-in would pass an invalid --client_env flag to Bazel? Shall we file an IJWB bug?

[brendandouglas]

No idea. The plugin doesn't do it directly, however perhaps it's coming in
via an environment variable or similar?

You can see the exact bazel invocation in the error message:

/home/ulfjack/Software/bazel info --tool_tag=ijwb:IDEA:community
--curses=no --color=no --noexperimental_ui
--noprogress_in_terminal_title --

It is suspicious that this has just started to happen, with no
corresponding plugin-side changes to the 'bazel info' invocation -- perhaps
something has regressed in Bazel 0.5.1?

On Wed, Jul 5, 2017 at 4:27 AM, Philipp Wollermann <[email protected]

wrote:

@brendandouglas https://github.com/brendandouglas Any idea why the
plug-in would pass an invalid --client_env flag to Bazel? Shall we file an
IJWB bug?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3196#issuecomment-313036815,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATahkeRRgoZFTDP3JjeWhyQzg4GkG0Deks5sK0jfgaJpZM4N6YQN
.

[ulfjack]

It's possible that there's a change in the Bazel client that makes us put bad env variable entries in the invocation (we copy the client env variables to a protobuf and send them over gRPC).

[harmank]

@ulfjack Can you please point to the patch you were talking about, to make intelliJ work for the time being?

[ulfjack]

Although, if the client does, it somehow needs to be triggered by the IJWB invocation, because we haven't seen any other reports of this problem.

[ulfjack]

https://github.com/ulfjack/bazel/commit/ab25e22f73e72d503470d76d24430da637a26478 should work around the issue. I didn't try it since it only happens on my other laptop, which I don't have with me right now.

[harmank]

@ulfjack On my machine ulfjack/bazel@ab25e22 fails to make intelliJ work. Can you please confirm on your side?

[ulfjack]

I can't reproduce it anymore. Are you sure you patched in the right change, and that IJ is using the patched Bazel binary? Do you get an error message?

[harmank]

@ulfjack I patched the right change. IJ is using the patched Bazel binary. I think this error is new. As earlier the command "bazel info" works fine in terminal but not in bazel console. I am using IntelliJ IDEA 2017.1.5 with bazel plugin version: 2017.06.05.0.4 on Ubuntu 16.04, if needed.

[harmank]

@ulfjack As suspected the new error was problem with my config only. Invalidating caches made it work. Thanks for the help :)

[ulfjack]

My patch is causing the NPE (issue #3331), so if anyone wants to do something like my patch, they'd have to take care of that.

[mmm444]

I have looked into this issue and found out that the cause is very likely a bug in OpenJDK that is causing environment corruption in the IntelliJ process. To my knowledge it is only happening on systems where the default Swing Look&Feel is GTK. I have submitted a bug report to Oracle. I will update this discussion once they respond.

The workaround that works for me is to unset GTK_MODULES environment variable before running IntelliJ.

[ulfjack]

Wow, well spotted. We could implement a workaround in Bazel if necessary. An OpenJDK bug seems unlikely to be fixed quickly. :-(

[mmm444]

The JDK bug is here https://bugs.openjdk.java.net/browse/JDK-8185325 and is to my surprise being worked on...

[mmm444]

The JDK bug has been classified as a security issue and is now fixed in Oracle Java 8u161 and 9.0.4. It even has its own CVE-2018-264. Now we only have to wait until JetBrains update the JDK bundled with IDEA.

[philwo]

@mmm444 Congrats for finding a security issue in the JDK and thank you for getting it fixed upstream!