Bazel: [build fail] python: can't open '....': [Errno2] No such file or directory

[sp4ke]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

Trying to build tensorflow with opencl support using this repo.
The bazel build is giving me the following error
python: can't open file 'external/local_config_sycl/crosstool/computecpp': [Errno 2] No such file or directory

Here's a verbose output:

ERROR: /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/external/farmhash_archive/BUILD.bazel:19:1: C++ compilation of rule '@farmhash_archive//:farmhash' failed (Exit 2): computecpp failed: error executing command 
  (cd /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/execroot/org_tensorflow && \
  exec env - \
    COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-0.3.1-Linux \
    HOST_CXX_COMPILER=/usr/bin/g++ \
    HOST_C_COMPILER=/usr/bin/gcc \
    PATH=/home/spike/.nvm/versions/node/v6.10.1/bin:/home/spike/.local/bin:/home/spike/bin:/home/spike/.pyenv/plugins/pyenv-virtualenv/shims:/home/spike/.pyenv/shims:/home/spike/scripts:/home/spike/projects/go:/home/spike/projects/go/bin:/home/spike/.local/bin:/home/spike/bin:/home/spike/.pyenv/plugins/pyenv-virtualenv/shims:/home/spike/.pyenv/shims:/home/spike/scripts:/home/spike/projects/go:/home/spike/projects/go/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/home/spike/.pyenv/versions/ml/bin/python \
    PYTHON_LIB_PATH=/home/spike/.pyenv/versions/ml/lib/python3.5/site-packages \
    TF_NEED_CUDA=0 \
    TF_NEED_OPENCL=1 \
    TF_VECTORIZE_SYCL=1 \
  external/local_config_sycl/crosstool/computecpp -fPIE -fno-omit-frame-pointer -Wall -msse3 -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' -MD -MF bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.o' -fPIC -iquote external/farmhash_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive/src -isystem external/bazel_tools/tools/cpp/gcc3 -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/farmhash_archive/src/farmhash.cc -o bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.o).
python: can't open file 'external/local_config_sycl/crosstool/computecpp': [Errno 2] No such file or directory
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 0.313s, Critical Path: 0.08s

When I check inside /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/execroot/org_tensorflow the path external/local_config_sycl/crosstool/computecpp exists and I can run it from the command line and I can even compile the file manually using the command that is run by bazel from the same working directory.

Inside the external path I changed all symlinks to real files but still the same error.

How can I debug the python command that is run by bazel in this case ?

If possible, provide a minimal example to reproduce the problem:

Any bazel build --config=sycl command

Environment info

• Operating System: Arch Linux 64

• Bazel version (output of bazel info release):
  both release 0.5.4- (@non-git) and development version (HEAD)

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web? No

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

[damienmg]

Sounds like sandboxing so @philwo to take a stab at it.

[Taltzipi]

I Ran into a similar issue compiling tensorflow using a pyenv python version, where I received

python: can't open file 'external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc': [Errno 2] No such file or directory

It seems there's a clash in the use of pyenv and bazel.

What worked for me was:

1. temporarily disable pyenv (commenting out its initialization in .bashrc and opening a new bash) so that 'which python' points to the system's /usr/bin/python,
2. run tensorflow's configure.py. When asked for the python version, point it to a specific pyenv python version you've installed, eg ~/.pyenv/versions/3.6.1/bin/python
3. continue with the bazel build - it should now run successfully.
4. reenable pyenv in bashrc

There's probably a more elegant way, but worth giving this a try.

[philwo]

Thanks for your suggestion @Taltzipi! I can imagine that this helps.

I don't think this is a sandboxing bug. Usually these kind of issues result from the environment being somehow "special" (e.g. Python is installed via pyenv, from source, relies on LD_LIBRARY_PATH or PATH being set to special values...).

Not saying that this shouldn't work with Bazel, but it means that I also don't have any more clue what the reason is than anyone else on the team ;)

It might help to consider that the error "[Errno 2] No such file or directory" can also (a bit confusingly) mean that the binary relies on shared libraries that were not found, which could be, because Bazel strips certain environment variables when running actions?

[vishnubob]

I am building tensorflow from scratch for the first time, and I immediately hit this issue. I did not want to restart the whole process over again without my carefully constructed pyenv environment, so I looked for a solution. Since the file external/local_config_sycl/crosstool/computecpp is located within our build cache, my (gross) fix was to add a new line to the top of this python script that points directly to the pyenv python executable I wish to run.

(echo "#\!$(pyenv which python)" && cat /PATH/TO/crosstool_wrapper_driver_is_not_gcc) > /tmp/cross
mv /tmp/cross /PATH/TO/crosstool_wrapper_driver_is_not_gcc

I believe the "problem" is the builders' usage of env - which dumps the environment before it hands the subshell over to external/local_config_sycl/crosstool/computecpp via exec. Since we are in a naked environment, I would assume the pyenv shims aren't properly initialized, and the shell reports that it can't find python.

[philwo]

Unassigning - sorry, I can't be of help fixing this. It's not really a sandboxing issue (in the sense of: we could change Bazel's sandbox somehow to make this work). I think pyenv might be just fundamentally incompatible with our sandboxing - but that said, I don't know anything about pyenv. I think someone with more experience in Python should look into this.