Bazel: Make bazel use a non system GCC on RHEL 6

[jeb2239]

Description of the problem / feature request:

How can I force the bazel runtime environment to use a specific LdLibrary Path ?
How can I force bazel to use a compiler installed in a non standard place?
Specifically I am trying to build some code on RHEL 6 and while I am able to use my newer compiler to compile the bazel tool when ever I try to run something it fails.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

On rhel 6 with a non standard location g++ the following will always happen.

../bazel_0.90/output/bazel run //:gazelle         
.......
INFO: Analysed target //:gazelle (182 packages loaded).
INFO: Found 1 target...
ERROR: /tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/external/io_bazel_rules_go/go/tools/builders/BUILD.bazel:33:1: error executing shell command: 'export GOROOT=$(pwd)/external/go_sdk && export GOROOT_FINAL=GOROOT && external/go_sdk/bin/go tool compile -trimpath $(pwd) -o bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_a...' failed (Exit 1): process-wrapper failed: error executing command 
  (cd /tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/execroot/__main__ && \
  exec env - \
    TMPDIR=/tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/bazel-sandbox/6357395096627452836/execroot/__main__/tmp \
  /tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/execroot/__main__/_bin/process-wrapper '--timeout=0' '--kill_delay=15' /bin/bash -c 'export GOROOT=$(pwd)/external/go_sdk && export GOROOT_FINAL=GOROOT && external/go_sdk/bin/go tool compile -trimpath $(pwd) -o bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile~/go/tools/builders/compile.a external/io_bazel_rules_go/go/tools/builders/compile.go external/io_bazel_rules_go/go/tools/builders/env.go external/io_bazel_rules_go/go/tools/builders/filter.go external/io_bazel_rules_go/go/tools/builders/flags.go')
/tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/execroot/__main__/_bin/process-wrapper: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /tmp/bazel/dd52caa7fc34a16b1ed8341a8fb75f2d/execroot/__main__/_bin/process-wrapper)
Target //:gazelle failed to build
INFO: Elapsed time: 5.744s, Critical Path: 0.61s
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target

What operating system are you running Bazel on?

RHEL 6

What's the output of bazel info release?

release 0.9.0- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

using the provided compile command ./compile.sh building the tool gives me no issues but it seems like when ever I try to run something the tool chain pics up an old system library instead of the one that goes with the compiler that was used to build it.

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

this was a release zip so there is no git info.

Have you found anything relevant by searching the web?

Yes several things with in these issues.

https://github.com/bazelbuild/bazel/issues/1420

https://github.com/bazelbuild/bazel/issues/1358 -- seems like this is most similar to my issue.

[hlopko]

You can set CC environment variable to point to the compiler to be used. Please reopen if you couldn't make it work. Thanks!

[29x10]

@mhlopko

I met the same error, go tool compile could not detect the correct libstdc++.so.6

I have set CC and LD_LIBRARY_PATH, I even build a special bazel with modified CROSSTOOL

[hlopko]

Looking into this now, it seems like I answered a wrong question, I don't see specific C++ issue here, more for go rules.

Jay, is this fixed now, should we reopen?

[jayconrod]

@mhlopko This was also opened as bazelbuild/rules_go#1766.

My interpretation of the error message is that process-wrapper has a C++ dependency the dynamic linker is not able to find. Earlier I thought process-wrapper was part of this project, but I guess it's actually part of Bazel (src/main/tools/process-wrapper.cc).

The Go executables don't have a C++ dependency, and I don't believe this problem is related to rules_go. It's just reported the first time Bazel runs a Go action with process-wrapper.

When was this fixed? I'm guessing the solution was to pipe LD_LIBRARY_PATH in somewhere?

@29x10 What version of Bazel are you using?

[29x10]

@jayconrod

I am using bazel 0.17.2, I fixed it by manually change the file PosixLocalEnvProvider.java in bazel project to set LD_LIBRARY_PATH

[LukeIreland1]

@jayconrod

I am using bazel 0.17.2, I fixed it by manually change the file PosixLocalEnvProvider.java in bazel project to set LD_LIBRARY_PATH

How did you do this?

[krafczyk]

I have the same problem. I'm trying to use bazel 0.26.1 to build tensorflow on a RHEL 6 system and have the same failure. I reported this initially as a bug with tensorflow here: tensorflow/tensorflow#32510, however it became clear that bazel's process-wrapper was the problem.

@jayconrad, I'd like to know how you modified PosixLocalEnvProvider.java to set this path as well. I'll let folks know if I find my own solution and what it was.

[krafczyk]

For folks encountering this issue, I wound up backporting the commit https://github.com/bazelbuild/bazel/commit/ab9c1f5f98a993b4da532224a9fed1b1b9a90148 to the version of bazel I needed to use (0.26.1).

Then setting BAZEL_LINKOPTS to "-static-libstdc++ -static-libgcc" and BAZEL_LINKLIBS to "-l%:libstdc++.a" caused bazel to be built without the c++ runtime shared linked.

[jgarvin]

The work around from @krafczyk doesn't work if you need to dynamically link the standard library. In my code base I have to link a library from a third party that has already chosen to link the standard library dynamically, so statically linking it myself will lead to a mix of versions (and indeed crashes with errors similar to this thread).

I'm a little flummoxed that the leading suggestion for telling bazel to link libstdc++ from a different location is to just not do it. Is nobody using bazel and building gcc themselves and has to use third party code? :P