bazel ignores LD_LIBRARY_PATH when building tensorflow

[geralt32]

I'm trying to build tensorflow on a RHEL 6.3 system with an older GLIBC_2.12, so I can't use the precompiled binaries which require GLIBC_2.16.

The system compiler is old, so we use another one (gnu/5.1.0) installed in /opt. The PATH, LD_LIBRARY_PATH, INCLUDE variables are set accordingly with the module environment tools.

I've built bazel from scratch with jdk/1.80_60 and gnu/5.1.0 without any issues.
Now I'm trying to build tensorflow with the modules jdk/1.8.0_60, gnu/5.1.0, python/3.4.3.
The first issue I've noticed was that PYTHONHOME was ignored but PYTHONPATH wasn't so I've set PYTHONPATH accordingly.

After running configure successfully, then "bazel build -s -c opt //tensorflow/tools/pip_package:build_pip_package" the build procedure aborts with this:

--snip--
>>>>> # //tensorflow/core/debug:debug_service_proto_cc_genproto [action 'ProtoCompile tensorflow/core/debug/debug_service.pb.cc'] (cd /projects/nierodal/src/tensorflow.build/execroot/tensorflow && \ exec env - \ bazel-out/host/bin/external/protobuf/protoc '--cpp_out=bazel-out/local-py3-opt/genfiles/' '--plugin=protoc-gen-grpc=bazel-out/host/bin/external/grpc/grpc_cpp_plugin' '--grpc_out=bazel-out/local-py3-opt/genfiles/' -I. -I. -Iexternal/protobuf/src -Ibazel-out/local-py3-opt/genfiles/external/protobuf/src -Iexternal/protobuf/src -Ibazel-out/local-py3-opt/genfiles/external/protobuf/src tensorflow/core/debug/debug_service.proto) ERROR: /projects/nierodal/src/tensorflow/tensorflow/python/BUILD:2148:1: null failed: protoc failed: error executing command bazel-out/host/bin/external/protobuf/protoc '--cpp_out=bazel-out/local-py3-opt/genfiles/' -I. -I. -Iexternal/protobuf/src -Ibazel-out/local-py3-opt/genfiles/external/protobuf/src ... (remaining 3 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by bazel-out/host/bin/external/protobuf/protoc) bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version 'CXXABI_1.3.8' not found (required by bazel-out/host/bin/external/protobuf/protoc) bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version 'GLIBCXX_3.4.18' not found (required by bazel-out/host/bin/external/protobuf/protoc)
--snap--

So it is obviously checking the system compiler libs (/usr/lib64/libstdc++) which are too old, but ignores the installed gnu/5.1.0 libs even though their path is set in the exported env variable LD_LIBRARY_PATH.

How can this info be passed to the compiler?

The bazel info is:

$ bazel info release
release 0.4.4- (@non-git)

[kchodorow]

You can modify the CROSSTOOL file, which defines where Bazel looks when building C++ targets. Take a look at the wiki for instructions.

[geralt32]

Haven't tried that yet. What worked was adding use_default_shell_env=True, to the protobuf.bzl file in the cache:
ctx.action( inputs=inputs, outputs=ctx.outputs.outs, arguments=args + import_flags + [s.path for s in srcs], executable=ctx.executable.protoc, mnemonic="ProtoCompile", use_default_shell_env=True, )

together with some hardcoded paths in the existing CROSSTOOL.tpl.
I'll try this solution next week, with a cleaned up code, without tweaks.

[kchodorow]

Oh yeah, use_default_shell_env is probaby easier. Let us know how it works out.

[dslomov]

@geralt32 were you able to solve this, either with use_default_shell_env=True, or by tweaking CROSSTOOL?

[geralt32]

@dslomov yes, I've tried to solve it with use_default_shell_env=True first and it worked. I've had to tweak a few things:

• replace the exec binary python with python3 in the tensorflow templates: crosstool_wrapper_driver_is_not_gcc.tpl, gen_git_source.py since our python3 installation doesn't link python to python3 itself
• add include directories of our custom compiler to tensorflow's CROSSTOOL.tpl
• add use_default_shell_env=True, to the ctx.action section in bazel's protobuf.bzl

I'll try the proposed toolchain solution later on.

[mharrend]

@geralt32 I can confirm that your proposal with adding

adding use_default_shell_env=True, to the protobuf.bzl file in the cache:
ctx.action( inputs=inputs, outputs=ctx.outputs.outs, arguments=args + import_flags + [s.path for s in srcs], executable=ctx.executable.protoc, mnemonic="ProtoCompile", use_default_shell_env=True, )

is working in case one has a non-default gcc installation path.

I have created the following PR to fix it upstream:
https://github.com/google/protobuf/pull/2933

[mharrend]

In case anyone is interested I am using the following Jenkinsfile to compile tensorflow:
https://github.com/mharrend/tensorflow-c--api/blob/master/Jenkinsfile

I will add a patch to add the modification to the protobuf.bzl file in a few minutes.

[OnlyBelter]

Who can tell me where is protobuf.bzl located?

[mharrend]

@OnlyBelter Look here for the file: protobuf/protobuf.bzl
https://github.com/google/protobuf/blob/master/protobuf.bzl

But be aware that Tensorflow uses a dedicated protobuf version. It can be found here in the Tensorflow folder: tensorflow/bazel-tensorflow/external/protobuf/protobuf.bzl

[da-steve101]

another work around is to remove ~/.cache/bazel
it then re-reads the paths when rebuilding the cache

[jin]

Closing as obsolete, and it looks like there are workarounds.

[andrewor14]

I just wanted to add that use_default_shell_env=True worked for me too, but I had to add this in a few different files. I was trying to build Tensorflow v2.2.0-rc4 with a custom gcc that I configured through the CC environment variable. In addition to adding this flag to protobuf.bzl in my bazel cache, I also had to add it to the ctx.actions.run call in tensorflow/tensorflow.bzl. I hope there's a smoother way to do this in the future.