Bazel: Building with ccache fails with auto-configuration

[akors]

Hi! I tried to recompile bazel, but I ran into "missing dependency declarations" errors:

ERROR: /home/alexander/source/bazel/src/BUILD:125:2 undeclared inclusion(s) in rule '//src/main/tools:process-wrapper':
this rule is missing dependency declarations for the following files included by 'src/main/tools/process-wrapper.c':
  '/usr/include/stdc-predef.h'
  '/usr/include/err.h'
  '/usr/include/features.h'
  '/usr/include/sys/cdefs.h'
....

The full output can be found here: https://gist.github.com/akors/16020dac060948e87c1d14a9794f12db

Note that this is very similar #715, HOWEVER, the directories listed are definitely present in my CROSSTOOL file.

$ grep cxx_builtin_include  tools/cpp/CROSSTOOL
  cxx_builtin_include_directory: "/usr/lib/gcc/"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/include"
  cxx_builtin_include_directory: "/"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/include"
...

I'm running on Fedora 23, and trying to get bazel to use a custom compiler - but I can't even get it to use my system compiler.

[damienmg]

Are you building from head?
gcc -E -xc++ -v returns what?

[akors]

I was using the master branch at 0b26f446f8312d1c43d162fe706467ef458c4db8, and now the issue still persists at the current HEAD 790d2f6009d47fe92cf0cd92a1473bbf0141f32e.

Here's what gcc -E -xc++ -v returns:

gcc: warning: '-x c++' after last input file has no effect
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --disable-libgcj --with-isl --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 5.3.1 20160406 (Red Hat 5.3.1-6) (GCC) 

---

By the way, I am not affected by this issue anymore, because I'm using version 0.2.1 + a20352eb1dce3fd8df2527ad7227abf5f1bb7891 , and I use my own self-compiled GCC 4.9.
This opened a whole different can of worms, and ploughing through issue #760 and trying to finally get bazel to work with a custom GCC installation made me a very sad puppy.
Being a good citizen, I should probably report my issues, but quite frankly I'm a bit tired. I just wanted to _use_ TensorFlow, but it turns out I need to change TensorFlow, and now I'm finding myself on a bug tracker for TensorFlow's build tool. Sorry for whining. Feel free to close.

[damienmg]

arf I got the syntax wrong, that was supposed to be gcc -E -xc++ - -v (note the extra -)

I understand your frustration, Bazel was designed originally for a very controlled environment (we vendor our C++ toolchain on the repository in Google). I have made some recent change so custom toolchain works out of the box but there are still some rough edges. Really sorry for that.

[akors]

I think you meant to pipe something in there ;)

$ echo '' | gcc -E -xc++ - -v 
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --disable-libgcj --with-isl --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 5.3.1 20160406 (Red Hat 5.3.1-6) (GCC) 
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'
 /usr/libexec/gcc/x86_64-redhat-linux/5.3.1/cc1plus -E -quiet -v -D_GNU_SOURCE - -mtune=generic -march=x86-64
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/5.3.1/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../x86_64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../include/c++/5.3.1/backward
 /usr/lib/gcc/x86_64-redhat-linux/5.3.1/include
 /usr/local/include
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "<command-line>" 2
# 1 "<stdin>"
COMPILER_PATH=/usr/libexec/gcc/x86_64-redhat-linux/5.3.1/:/usr/libexec/gcc/x86_64-redhat-linux/5.3.1/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/5.3.1/:/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/x86_64-redhat-linux/5.3.1/:/usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/5.3.1/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-mtune=generic' '-march=x86-64'

No worries. I understand that fresh software can be quite rough around the edges. And I really, really appreciate you folks (as in Google) putting your precious code out there in the wild, and you Developers actually tending to it. It's just... hard sometimes.

[damienmg]

Sorry this bug went under my radar. Can you tell what which gcc returns? Do you have the CC environment variable set? If yes to where?

[akors]

Don't worry about it. I'm using 0.2.1 which works, so this bug has low priority for me too. I'm still available to help you fix it though.

$ which gcc
/usr/lib64/ccache/gcc

CC is not set.

[akors]

When I set CC=/usr/bin/gcc and CXX=/usr/bin/g++ , the compilation succeeds with the latest HEAD (8b388a0739ffaff1487911116f78a4a97004df6e), but not with the previous one.

Without explicitly setting the CC and CXX variables, the compilation still fails like before.

[damienmg]

Ok ccache is the problem with latest version of Bazel it tries to detect gcc, so uses your ccache version which does fails because ccache has not access to previous build in Bazel, that's why setting CC correctly fix the issue.

This is working as intended. See #1139 for more information

[akors]

This is working as intended. See #1139 for more information

Ok, so I'm closing this issue?

I suggest that you put this piece of information somewhere visible (like in a FAQ) or something, because on some modern linux distributions, GCC is by default set to use ccache.

[damienmg]

I agree with you so reopening for correctly documenting that issue, maybe we can even try to find gcc behind ccache instead.

[damienmg]

Ok the problems is with installation of ccache, so we should just document that issue as a standard one. ccache use a different gcc that /usr/bin/gcc

[hhclam]

Mind if I ask. What distribution uses ccache by default in place of gcc?

[akors]

Mind if I ask. What distribution uses ccache by default in place of gcc?

Ok, apperently it's only Fedora. Not Debian, Ubuntu, Arch or Centos. Did not check any others.

However, it seems that my closing the issue was a bit premature. Compilation now fails, even when I set CC=/usr/bin/gcc and CXX=/usr/bin/g++ . It succeeds however, when I set CC=/opt/gcc-4.9/bin/gcc, this being my self-compiled GCC. This is getting really strange. I was pretty darn sure that my system-gcc was succesfully able to compile bazel... The current head for me is now 0ba25552dc4a640c2616394975dca9aaa724fb92 .

[damienmg]

For the CC issue, I got it tracked down in #1152 too. I think to make it workaround when ccache is there, we could use the COLLECT_GCC_OPTIONS answer from gcc -E

[antonovvk]

Hi!
I fixed the issue (in Fedora 25) by configuring ccache:
ccache -o path=$PATH -o base_dir=$HOME