Bazel: "Invalid EvalException" from Bazel when building Tensorflow on Mac OS X

Created on 25 Jan 2017  路  8Comments  路  Source: bazelbuild/bazel

See Tensorflow issue:
https://github.com/tensorflow/tensorflow/issues/7048

The original reporter (@yaroslavvb) gave this error message.

Log:
bash-3.2$ ./configure
./configure
Please specify the location of python. [Default is /Users/yaroslav/anaconda/bin/python]: 

Please specify optimization flags to use during compilation [Default is -march=native]: 

Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] 

No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N] 

No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N] y
y
XLA JIT support will be enabled for TensorFlow
Found possible Python library paths:
  /Users/yaroslav/anaconda/lib/python3.5/site-packages
Please input the desired Python library path to use.  Default is [/Users/yaroslav/anaconda/lib/python3.5/site-packages]


Using python library path: /Users/yaroslav/anaconda/lib/python3.5/site-packages
Do you wish to build TensorFlow with OpenCL support? [y/N] 

No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] y
y
CUDA support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 

Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 

Please specify the location where CUDA  toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 

Please specify the Cudnn version you want to use. [Leave empty to use system default]: 

Please specify the location where cuDNN  library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 

libcudnn.dylib resolves to libcudnn.dylib
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 

____Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
.........
____Loading package: tensorflow/contrib/image
____Loading package: tensorflow/core/debug
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 65,536 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 1,676,424 bytes
____Loading package: @bazel_tools//tools/jdk
____Loading package: tools/defaults
____Loading package: third_party/py/numpy
____Loading package: util/python
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@llvm' (requested by nodes 'REPOSITORY:@llvm')
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid EvalException:
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
    at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:196)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryContext.downloadAndExtract(SkylarkRepositoryContext.java:594)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:316)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
    at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
    at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
    at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
    at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
    at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
    at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
    at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    at com.google.devtools.build.lib.syntax.EvalException.<init>(EvalException.java:112)
    at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:209)
    at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:217)
    at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:344)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
    at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
    at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
    at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
    at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
    at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
    at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
    at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
    ... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@llvm' (requested by nodes 'REPOSITORY:@llvm')
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Invalid EvalException:
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
    at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:196)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryContext.downloadAndExtract(SkylarkRepositoryContext.java:594)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:316)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
    at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
    at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
    at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
    at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
    at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
    at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
    at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:501)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    at com.google.devtools.build.lib.syntax.EvalException.<init>(EvalException.java:112)
    at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:209)
    at com.google.devtools.build.lib.syntax.EvalException$EvalExceptionWithJavaCause.<init>(EvalException.java:217)
    at com.google.devtools.build.lib.syntax.FuncallExpression.callMethod(FuncallExpression.java:344)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:732)
    at com.google.devtools.build.lib.syntax.FuncallExpression.invokeObjectMethod(FuncallExpression.java:784)
    at com.google.devtools.build.lib.syntax.FuncallExpression.doEval(FuncallExpression.java:770)
    at com.google.devtools.build.lib.syntax.Expression.eval(Expression.java:48)
    at com.google.devtools.build.lib.syntax.ExpressionStatement.doExec(ExpressionStatement.java:46)
    at com.google.devtools.build.lib.syntax.Statement.exec(Statement.java:37)
    at com.google.devtools.build.lib.syntax.UserDefinedFunction.call(UserDefinedFunction.java:136)
    at com.google.devtools.build.lib.syntax.BaseFunction.call(BaseFunction.java:439)
    at com.google.devtools.build.lib.bazel.repository.skylark.SkylarkRepositoryFunction.fetch(SkylarkRepositoryFunction.java:106)
    at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
    ... 4 more

(I have not reproduced this failure personally.)

Any clues?

P2 bug
Source
picture hawkinsp

All 8 comments

+1

sneakers-the-rat picture sneakers-the-rat on 26 Jan 2017

+1

roski picture roski on 27 Jan 2017

I saw this with 0.4.2, this doesn't happen to me with head. Any thoughts @jart?

kchodorow picture kchodorow on 27 Jan 2017
👍1

This is a duplicate of https://github.com/bazelbuild/bazel/issues/2232 and was solved in https://github.com/bazelbuild/bazel/commit/b79e2f689a58d2943ed84c9bcebfb7c538ad6b07 which unfortunately didn't make it into the 0.4.3 release.

I believe this error only occurs when there's a completely unrelated error. What happens is Bazel tries to shut down all the downloader threads, in order to display the other error. But they aren't able to shut down correctly, because a repository_rule is using download_and_extract.

TensorFlow has been running into this error a lot. So we're really looking forward to a release that includes the aforementioned commit.

jart picture jart on 27 Jan 2017
👍1

We'll have the release of 0.4.4 this week, so hopefully this will be fixed by then.

meteorcloudy picture meteorcloudy on 30 Jan 2017
👍1

Thanks for the info y'all :)

sneakers-the-rat picture sneakers-the-rat on 31 Jan 2017

We just released 0.4.4, hope it can help everyone encountering this problem! :)
https://github.com/bazelbuild/bazel/releases/tag/0.4.4

meteorcloudy picture meteorcloudy on 2 Feb 2017
👍1

As far as I am aware, this issue is now fixed.

hawkinsp picture hawkinsp on 6 Feb 2017
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ob picture ob  路  3Comments
cyberbono3 picture cyberbono3  路  3Comments
meteorcloudy picture meteorcloudy  路  3Comments
ensonic picture ensonic  路  3Comments
filipesilva picture filipesilva  路  3Comments