Bazel: IllegalStateException: Did not write as much data as expected

Created on 12 Sep 2018 · 16Comments · Source: bazelbuild/bazel

We've seen the following failure cause 2 flakes on our CI server (out of around 30 builds since upgrading from Bazel 0.15.0 to Bazel 0.16.1).

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
INFO: Elapsed time: 178.732s, Critical Path: 6.46s
INFO: 186 processes: 186 remote cache hit.
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false, actionIndex=0}' (requested by nodes 'external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/_ijar/jar/external/com_fasterxml_jackson_dataformat_jackson_dataformat_yaml/jar/jackson-dataformat-yaml-2.9.0-ijar.jar @com_fasterxml_jackson_dataformat_jackson_dataformat_yaml//jar:jar BuildConfigurationValue.Key[56993b598966274b11f9c239ecf81d3c] false')
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
    at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:973)
    at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
    at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
    at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
    at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
    at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
    at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:90)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:63)
    at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
    at com.google.devtools.build.lib.analysis.actions.SpawnAction.internalExecute(SpawnAction.java:287)
    at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:294)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:978)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:910)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$900(SkyframeActionExecutor.java:120)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:457)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:513)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:227)
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:400)
    ... 4 more

I haven't seen it occur on my workstation - only on the CI build.

What operating system are you running Bazel on?

Ubuntu 14.04 host, running Bazel inside a docker container based on debian:jessie-backports.

What's the output of `bazel info release`?

0.16.1

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/rules_scala/issues/589 seems to be the same issue.

P2 team-Remote-Exec bug

Source

drigz

All 16 comments

This is a rather disturbing error that looks like either an internal protobuf bug or a HotSpot bug. I'm leaning towards that latter, since it appeared with the embedded Java 9 upgrade. --host_jvm_args=-XX:-InlineUnsafeOps seems to make the problem go away.

benjaminp on 28 Sep 2018

Sorry to chime in, but we've been seeing this too. I suspect it could also be that the cache is poisoned and somehow a partial object was written to the cache? And when it is read back, this assertion is thrown.

jgavris on 28 Sep 2018

If it helps any others: I applied the workaround by adding this like to .bazelrc:

startup --host_jvm_args=-XX:+UnlockDiagnosticVMOptions --host_jvm_args=-XX:-InlineUnsafeOps

drigz on 30 Sep 2018

The workaround seems to not apply everywhere:

Starting local Bazel server and connecting to it...
.INFO: Reading 'startup' options from /path/to/.bazelrc: --host_jvm_args=-XX:+UnlockDiagnosticVMOptions --host_jvm_args=-XX:-InlineUnsafeOps
Server crashed during startup. Now printing /home/kbuilder/.cache/bazel/_bazel_kbuilder/7b053e52cb9de0c352841a1b58c652ee/server/jvm.out
Error: VM option 'InlineUnsafeOps' is develop and is available only in debug version of VM.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

ensonic on 19 Oct 2018

Sorry I was too quick. This particular build was still using bazel 0.15.0.

ensonic on 19 Oct 2018

We just ran into this on our CI with bazel 0.18.0.

https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70245/pull-kubernetes-e2e-gce-device-plugin-gpu/51169/

W1025 17:56:49.640] Target //:push-build failed to build
W1025 17:56:49.642] Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.642]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.643]     at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.643]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.643]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.643]     at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.643] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.643]     at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.644]     at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.644]     at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.644]     at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.644]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.644]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.644]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.645]     at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.646]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.646]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.646]     at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.646]     at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.646]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.646]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.647]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.647]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.647]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.647]     at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.647]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.648]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.648]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.648]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.648]     ... 4 more
W1025 17:56:49.648] 
W1025 17:56:49.652] INFO: Elapsed time: 77.314s, Critical Path: 29.61s
W1025 17:56:49.654] INFO: 362 processes: 355 remote cache hit, 7 linux-sandbox.
W1025 17:56:49.659] FAILED: Build did NOT complete successfully
W1025 17:56:49.663] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.666]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.666]     at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.666]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.667]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.667]     at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.667] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.667]     at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.668]     at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.668]     at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.668]     at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.668]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.668]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.669]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.670]     at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.670]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.670]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.671]     at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.671]     at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.671]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.671]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.672]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.672]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.672]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.672]     at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.672]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.673]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.673]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.673]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.673]     ... 4 more
W1025 17:56:49.674] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.674]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.674]     at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.674]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.675]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.675]     at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.675] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.675]     at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.675]     at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.676]     at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.676]     at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.676]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.676]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.677]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.678]     at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.678]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.679]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.679]     at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.679]     at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.679]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.680]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.680]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.680]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.680]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.680]     at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.681]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.681]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.681]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.681]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.682]     ... 4 more
W1025 17:56:49.682] Remote logging disabled for testing, forcing abrupt shutdown.
W1025 17:56:49.683] com.google.devtools.build.lib.util.LoggingUtil#logToRemote: bazel crashed with args: run --startup_time=10 --restart_reason=no_daemon --binary_path=/usr/local/bin/bazel --rc_source=client --default_override=0:common=--isatty=0 --default_override=0:common=--terminal_columns=80 --rc_source=/etc/bazel.bazelrc --rc_source=/go/src/k8s.io/kubernetes/build/root/.bazelrc --default_override=1:build=--experimental_remote_spawn_cache --default_override=1:build=--remote_local_fallback --default_override=1:build=--remote_http_cache=http://bazel-cache.default.svc.cluster.local.:8080/kubernetes/kubernetes,de0d6280eabe96c208a99ad636f792c2 --default_override=2:test:integration=--local_test_jobs --default_override=2:test:integration=4 --default_override=2:test:integration=--test_tag_filters=integration --default_override=2:test:unit=--features=race --default_override=2:test:unit=--test_tag_filters=-e2e,-integration --default_override=2:test:unit=--flaky_test_attempts=3 --default_override=2:test=--test_output=errors --default_override=2:build:unit=--features=race --default_override=2:build=--verbose_failures --default_override=2:build=--workspace_status_command --default_override=2:build=hack/print-workspace-status.sh --default_override=2:build=--sandbox_tmpfs_path=/tmp --default_override=2:build=--sandbox_fake_username --client_cwd=/go/src/k8s.io/kubernetes //:push-build /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.13.0-alpha.2.65+4992a5afe427fe gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce-device-plugin-gpu/v1.13.0-alpha.2.65+4992a5afe427fe
W1025 17:56:49.684] java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=//vendor/github.com/mindprince/gonvml:go_default_library%linux_amd64%cgo_c_lib BuildConfigurationValue.Key[3190fc7906a0bb10bcf73823d08c1469] false, actionIndex=4}' (requested by nodes 'File:[[<execution_root>]bazel-out/k8-fastbuild/bin]vendor/github.com/mindprince/gonvml/libgo_default_library%linux_amd64%cgo_c_lib.lo')
W1025 17:56:49.684]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
W1025 17:56:49.684]     at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
W1025 17:56:49.684]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
W1025 17:56:49.684]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
W1025 17:56:49.685]     at java.base/java.lang.Thread.run(Unknown Source)
W1025 17:56:49.685] Caused by: java.lang.IllegalStateException: Did not write as much data as expected.
W1025 17:56:49.685]     at com.google.protobuf.CodedOutputStream.checkNoSpaceLeft(CodedOutputStream.java:977)
W1025 17:56:49.685]     at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:73)
W1025 17:56:49.685]     at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:85)
W1025 17:56:49.686]     at com.google.devtools.build.lib.remote.TreeNodeRepository.getOrComputeDirectory(TreeNodeRepository.java:391)
W1025 17:56:49.686]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:411)
W1025 17:56:49.686]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.686]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.TreeNodeRepository.computeMerkleDigests(TreeNodeRepository.java:409)
W1025 17:56:49.687]     at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:101)
W1025 17:56:49.688]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:101)
W1025 17:56:49.688]     at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:75)
W1025 17:56:49.688]     at com.google.devtools.build.lib.exec.SpawnActionContextMaps$ProxySpawnActionContext.exec(SpawnActionContextMaps.java:362)
W1025 17:56:49.688]     at com.google.devtools.build.lib.rules.cpp.CppLinkAction.execute(CppLinkAction.java:315)
W1025 17:56:49.688]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeActionTask(SkyframeActionExecutor.java:994)
W1025 17:56:49.688]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.prepareScheduleExecuteAndCompleteAction(SkyframeActionExecutor.java:923)
W1025 17:56:49.688]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.access$800(SkyframeActionExecutor.java:121)
W1025 17:56:49.688]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:763)
W1025 17:56:49.689]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.call(SkyframeActionExecutor.java:718)
W1025 17:56:49.689]     at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
W1025 17:56:49.689]     at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:471)
W1025 17:56:49.689]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:505)
W1025 17:56:49.689]     at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:215)
W1025 17:56:49.689]     at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:418)
W1025 17:56:49.689]     ... 4 more
W1025 17:56:50.300] 
W1025 17:56:50.300] Server terminated abruptly (error code: 14, error message: '', log file: '/bazel-scratch/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/server/jvm.out')

ixdy on 25 Oct 2018

For what it is worth, we are seeing recurrence of this error on our CI builds as well, with version 0.18.10.

sdqali on 16 Nov 2018

We got the same error in our CI builds using bazel 0.20.0

burkpojken on 19 Dec 2018

We have seen this on macOS CI with 0.22.0

keith on 14 Feb 2019

Presumably this will be resolved when 8db50b64b9600e392c3aa4644fdd87b2f2587ba2 is released with 0.23.

On Thu, Feb 14, 2019, at 14:03, Keith Smiley wrote:

We have seen this on macOS CI with 0.22.0

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/6136#issuecomment-463818465, or mute the thread https://github.com/notifications/unsubscribe-auth/AANZTn4Evt8zMybc0gcBRdbnUsiW41taks5vNd1FgaJpZM4WlW9y.

benjaminp on 14 Feb 2019

👍1

We haven't seen it on 0.22 since applying the recommended workaround here https://github.com/bazelbuild/bazel/issues/6136#issuecomment-425710997

jgavris on 15 Feb 2019

👍1

Seeing in 0.20.0 in Linux pipeline.

mariusgrigoriu on 9 Mar 2019

Unfortunately the workaround in
https://github.com/bazelbuild/bazel/issues/6136#issuecomment-425710997
breaks the build on macos x.

ensonic on 23 May 2019

@ensonic what version of bazel are you using? We haven't seen it on 0.22 and later on macOS.

jgavris on 23 May 2019

I wonder if this can be closed. Haven't seen this in ages.

mariusgrigoriu on 18 Nov 2020

Seems reasonable. I'll remove the workaround from our CI build and reopen if it recurs with a recent Bazel version.

drigz on 19 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Allow passing extra HTTP headers to remote cache

ob · 3Comments

Cannot define string literals in cc_library

f1recracker · 3Comments

Targets and packages can't be excluded with except for query and cquery deps

lisaonduty · 3Comments

repository_ctx.symlink on a large directory take a lot of time

alexandrvb · 3Comments

Support generating IntelliJ project files

davidzchen · 3Comments