Bazel: Windows: Unrecoverable error while evaluating node in SpawnAction

Created on 31 Aug 2017  Â·  11Comments  Â·  Source: bazelbuild/bazel

I'm trying to port GWT UI compilation (gwt.bzl) of Gerrit Code Review project to Windows. Note, that this Skylark rule works as expected on Linux and Mac Os X as is.

On Windows we have 2 unrelated problems with the code:

  • Classpath construction is not portable, as ":" is hard codes as classpath delimiter. I raised this issue #2045 to expose os attribute in context. However, it turned out, that ctx.configuration.host_path_separator already included in the context.
  • Classpath is appeared to be too long and cannot be passed as is, so that using the right classpath separator only reveals the next problem, that it cannot basically work as is (one single JAVA command line) and needs a different approach.

To fix the both problems, I adjusted the Skylark rule to copy (ln -s didn't work, obviously) all needed files in one directory and use JDK wildcard classpath feature to pass all JAR at once to the java proccess.

One complication here: dozen Gerrit GWT UI intermediate lilbraries have the final artifact name: libclient.jar, that located in different top-level directories, so that just copying them with only base name would overwrite some of them. So that we use the parent directory as well.

I ended up with this diff to most recent gerrit master to reproduce the problem: [1]. Running it exposes the next breakage in Bazel: [2].

$ bazel --nomaster_bazelrc build --verbose_failures -s gerrit-gwtui:ui_dbg.zip
....................
____Loading package: gerrit-gwtui
[...]
Target //gerrit-gwtui:ui_dbg.zip failed to build
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ACTION_EXECUTION:ActionLookupData{actionLookupNode=CONFIGURED_TARGET://gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (176401396 1938099534), actionIndex=1}' (requested by nodes 'gerrit-gwtui/ui_dbg.zip //gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (1350696638 1938099534)')
____Elapsed time: 29,181s, Critical Path: 10,39s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ACTION_EXECUTION:ActionLookupData{actionLookupNode=CONFIGURED_TARGET://gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (176401396 1938099534), actionIndex=1}' (requested by nodes 'gerrit-gwtui/ui_dbg.zip //gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (1350696638 1938099534)')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:413)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IndexOutOfBoundsException: position (2) must be less than the number of elements that remained (2)
        at com.google.common.collect.Iterators.get(Iterators.java:769)
        at com.google.common.collect.Iterables.get(Iterables.java:773)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:283)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:850)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:793)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:107)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:661)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:618)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:404)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:440)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:201)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:338)
        ... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ACTION_EXECUTION:ActionLookupData{actionLookupNode=CONFIGURED_TARGET://gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (176401396 1938099534), actionIndex=1}' (requested by nodes 'gerrit-gwtui/ui_dbg.zip //gerrit-gwtui:ui_dbg e2a2c30148df1102406ac458c6a4e154 (1350696638 1938099534)')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:413)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IndexOutOfBoundsException: position (2) must be less than the number of elements that remained (2)
        at com.google.common.collect.Iterators.get(Iterators.java:769)
        at com.google.common.collect.Iterables.get(Iterables.java:773)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:283)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:850)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:793)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:107)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:661)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:618)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:404)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:440)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:201)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:338)
        ... 4 more

The generated script file bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh is here: [3].

To demonstrate, that the problem with adjusted Skylark rule is Windows related, I applied the same patch on my Linux workstation, and the same rule works as expected: [4].

P2 area-Windows windows team-XProduct bug
Source
picture davido

All 11 comments

So looking at the offending source place SpawnAction.java:283:

String failMessage;
      if (isShellCommand()) {
        // The possible reasons it could fail are: shell executable not found, shell
        // exited non-zero, or shell died from signal.  The first is impossible
        // and the second two aren't very interesting, so in the interests of
        // keeping the noise-level down, we don't print a reason why, just the
        // command that failed.
        //
        // 0=shell executable, 1=shell command switch, 2=command
        try {
          failMessage =
              "error executing shell command: "
                  + "'"
                  + truncate(Iterables.get(argv.arguments(), 2), 200)
                  + "'";
        } catch (CommandLineExpansionException commandLineExpansionException) {
          failMessage =
              "error executing shell command, and error expanding command line: "
                  + commandLineExpansionException;
        }

reveals, that the command is failing and Bazel crashes in trying to dump the error message and makes some assumptions:

Iterables.get(argv.arguments(), 2)

that leads to IOOBE.

davido picture davido on 1 Sep 2017

//CC @tbroyer

I adjusted the Bazel error code in SpawnAction.java and now I'm seeing the actual error:

C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:114)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:261)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:268)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:850)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:793)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:107)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:661)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:618)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:404)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:440)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:201)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:338)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
ERROR: C:/users/davido/projects/gerrit/gerrit-gwtui/BUILD:10:1: error executing shell command: 'bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh' failed (Exit 126): bash.exe failed: error executing command
  cd C:/msys64/tmp/_bazel_davido/_biltsqa/execroot/gerrit
C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh.
[ERROR] Unable to create compiler work directory
java.io.IOException: Zugriff verweigert
        at java.io.WinNTFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)
        at com.google.gwt.util.tools.Utility.makeTemporaryDirectory(Utility.java:199)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:169)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:155)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:144)
        at com.google.gwt.dev.Compiler$1.run(Compiler.java:118)
        at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55)
        at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50)
        at com.google.gwt.dev.Compiler.main(Compiler.java:125)
Sep 01, 2017 8:50:27 AM java.util.prefs.WindowsPreferences <init>
davido picture davido on 1 Sep 2017

Intrestingly, when I cd to the execroot, and start te generated script directoy, it just works:

$ cd  C:/msys64/tmp/_bazel_davido/_biltsqa/execroot/gerrit

davido@DESKTOP-N3CDIH8 MSYS /tmp/_bazel_davido/_biltsqa/execroot/gerrit
$ bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
Sep 01, 2017 8:53:32 AM java.util.prefs.WindowsPreferences <init>
WARNUNG: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
Compiling module com.google.gerrit.GerritGwtUI
   Computing all possible rebind results for 'com.google.gerrit.client.ui.FancyFlexTableImpl'
      Rebinding com.google.gerrit.client.ui.FancyFlexTableImpl
         Could not find an exact match rule. Using 'closest' rule <replace-with class='com.google.gerrit.client.ui.FancyFlexTableImplIE8'/> based on fall back values. You may need to implement a specific binding in case the fall back behavior does not replace the missing binding
   Compiling 5 permutations
      Compiling permutation 0...
      Compiling permutation 1...
      Compiling permutation 2...
      Compiling permutation 3...
      Compiling permutation 4...
   Compile of permutations succeeded
   Compilation succeeded -- 39,578s
Linking into C:\msys64\tmp\_bazel_davido\_biltsqa\execroot\gerrit\bazel-out\msvc_x64-fastbuild\bin\gerrit-gwtui\ui_dbg.zip.gwt_output\gerrit_ui
   Link succeeded
   Linking succeeded -- 6,734s
File . does not seem to exist.
davido picture davido on 1 Sep 2017

If i pass valid working directory to GWT compiler:

[...] -workDir bazel-out/msvc_x64-fastbuild/bin/gerrit-gwtui/ui_dbg.zip.gwt_working [...]

then it is failing later with another attempt to do asomething with TMP directory:

C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
com.google.devtools.build.lib.exec.SpawnExecException: bash.exe failed: error executing command
  cd C:/msys64/tmp/_bazel_davido/_biltsqa/execroot/gerrit
C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:114)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:261)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:268)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:850)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:793)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:107)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:661)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:618)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:404)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:440)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:201)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:338)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh
ERROR: C:/users/davido/projects/gerrit/gerrit-gwtui/BUILD:10:1: error executing shell command: 'bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh' failed (Exit 126): bash.exe failed: error executing command
  cd C:/msys64/tmp/_bazel_davido/_biltsqa/execroot/gerrit
C:/msys64/usr/bin/bash.exe -c bazel-out/msvc_x64-fastbuild/genfiles/gerrit-gwtui/ui_dbg.run_shell_0.sh.
Compiling module com.google.gerrit.GerritGwtUI
[ERROR] Unexpected internal compiler error
java.lang.ExceptionInInitializerError
        at com.google.gwt.dev.javac.CompiledClass.<clinit>(CompiledClass.java:41)
        at com.google.gwt.dev.javac.JdtCompiler$CompilerImpl.createCompiledClass(JdtCompiler.java:405)
        at com.google.gwt.dev.javac.JdtCompiler$CompilerImpl.process(JdtCompiler.java:368)
        at org.eclipse.jdt.internal.compiler.Compiler.processCompiledUnits(Compiler.java:546)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:458)
        at com.google.gwt.dev.javac.JdtCompiler.doCompile(JdtCompiler.java:1092)
        at com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.compile(CompilationStateBuilder.java:325)
        at com.google.gwt.dev.javac.CompilationStateBuilder.doBuildFrom(CompilationStateBuilder.java:548)
        at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:479)
        at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:465)
        at com.google.gwt.dev.cfg.ModuleDef.getCompilationState(ModuleDef.java:423)
        at com.google.gwt.dev.Precompile.precompile(Precompile.java:222)
        at com.google.gwt.dev.Precompile.precompile(Precompile.java:202)
        at com.google.gwt.dev.Precompile.precompile(Precompile.java:143)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:204)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:155)
        at com.google.gwt.dev.Compiler.compile(Compiler.java:144)
        at com.google.gwt.dev.Compiler$1.run(Compiler.java:118)
        at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55)
        at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50)
        at com.google.gwt.dev.Compiler.main(Compiler.java:125)
Caused by: java.lang.RuntimeException: Unable to initialize byte cache
        at com.google.gwt.dev.util.DiskCache.<init>(DiskCache.java:61)
        at com.google.gwt.dev.util.DiskCache.<clinit>(DiskCache.java:49)
        ... 21 more
Caused by: java.io.IOException: Zugriff verweigert
        at java.io.WinNTFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)
        at java.io.File.createTempFile(File.java:2070)
        at com.google.gwt.dev.util.DiskCache.<init>(DiskCache.java:56)
        ... 22 more
davido picture davido on 1 Sep 2017

This is the offending part und GWT code:

https://github.com/gwtproject/gwt/blob/master/dev/core/src/com/google/gwt/dev/util/DiskCache.java#L54-L63

private DiskCache() {
    try {
      File temp = File.createTempFile("gwt", "byte-cache");
      temp.deleteOnExit();
      file = new RandomAccessFile(temp, "rw");
      file.setLength(0);
    } catch (IOException e) {
      throw new RuntimeException("Unable to initialize byte cache", e);
    }
  }
davido picture davido on 1 Sep 2017

Adjust the temp dir using -Djava.io.tmpdir=…?

Re. your "classpath too long", couldn't you also create a "fat jar" of all the dependencies too to avoid filename conflicts?

tbroyer picture tbroyer on 1 Sep 2017

Adjust the temp dir using -Djava.io.tmpdir=…?

Thanks, will give it a try and let you know.

Re. your "classpath too long", couldn't you also create a "fat jar" of all the dependencies too to avoid filename conflicts?

Good point, but currently we don't create that "uber jar". Note that GWT user and dev are already dozens of MBs and this would take significant time.

davido picture davido on 1 Sep 2017

@tbroyer

Adjust the temp dir using -Djava.io.tmpdir=…?

Yes, this fixed it. Strange, that gwt compiler is happy even without passing -Djava.io.tmpdir=foo on sane platforms.

davido picture davido on 5 Sep 2017

Sounds like it's https://github.com/bazelbuild/bazel/issues/2870 again. On Windows Bazel should set a meaningful TMP and TEMP value for every action. I have a design doc on how to do that, but I haven't published it yet.

laszlocsomor picture laszlocsomor on 5 Sep 2017

...Motivation: GetTempPath's behavior:

The GetTempPath function checks for the existence of environment variables in the following order and uses the first path found:

  • The path specified by the TMP environment variable.
  • The path specified by the TEMP environment variable.
  • The path specified by the USERPROFILE environment variable.
  • The Windows directory.
laszlocsomor picture laszlocsomor on 5 Sep 2017
❤1

//CC @aehlig

OK, it seems that underlying issue of missing TMP is a duplicate of: #1590, #2349, #2870.

Still the question is of UX failure to produce a sane error message that ends up with IndexOutOfBoundsException.

davido picture davido on 7 Sep 2017
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

C00IHandLuke picture C00IHandLuke  Â·  3Comments
ensonic picture ensonic  Â·  3Comments
filipesilva picture filipesilva  Â·  3Comments
iirina picture iirina  Â·  3Comments
f1recracker picture f1recracker  Â·  3Comments