Bazel: Blaze ran out of memory and crashed.

Created on 12 Feb 2016  ·  27Comments  ·  Source: bazelbuild/bazel

Hello,

I'm trying to build bazel from source on a farm server running CentOS-5.7. I don't have root access on this machine. My libc version 2.5. Here are the steps I take:

$) git clone https://github.com/bazelbuild/bazel.git
$) export PATH=/home/utils/gcc-4.8.4/bin:$PATH 
$) export LD_LIBRARY_PATH=/home/utils/gcc-4.8.4/lib64:$LD_LIBRARY_PATH 
$) export JAVA_HOME=/home/utils/java/jdk1.8.0_51/
$) ./compile.sh

Here is the output I observe:

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
.INFO: Elapsed time: 0.667s
unable to create new native thread

Blaze ran out of memory and crashed.

Building output/bazel

I'd appreciate any suggestions on how to fix this or how to dump out more information about this crash.

My eventual goal is is build TensorFlow from source using bazel.

Thank you!

under investigation
Source
picture ghost

Most helpful comment

It worked for me by setting --jobs 10 and --ram_utilization_factor 25.
Basically i am trying to build the tensorflow python pip package from source, so the command now becomes,

bazel build --jobs 10 --ram_utilization_factor 25 -c opt //tensorflow/tools/pip_package:build_pip_package

I think this is happening cause of the poor specs of the machine.

picture sushantMoon on 5 Jul 2016
❤1 👍1

All 27 comments

This was also reported in https://github.com/bazelbuild/bazel/issues/660.
You can try reducing the number of jobs Bazel uses: add --jobs=10, for
instance. By default Bazel uses 200 threads.

On Fri, Feb 12, 2016 at 1:24 PM, snazzman [email protected] wrote:

Hello,

I'm trying to build bazel from source on a farm server running CentOS-5.7.
I don't have root access on this machine. My libc version 2.5. Here are the
steps I take:

$) git clone https://github.com/bazelbuild/bazel.git
$) export PATH=/home/utils/gcc-4.8.4/bin:$PATH
$) export LD_LIBRARY_PATH=/home/utils/gcc-4.8.4/lib64:$LD_LIBRARY_PATH
$) export JAVA_HOME=/home/utils/java/jdk1.8.0_51/
$) ./compile.sh

Here is the output I observe:

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO: ./compile.sh compile /path/to/bazel🍃 Building Bazel from scratch......🍃 Building Bazel with Bazel.
.INFO: Elapsed time: 0.667s
unable to create new native thread

Blaze ran out of memory and crashed.

Building output/bazel

I'd appreciate any suggestions on how to fix this or how to dump out more
information about this crash.

My eventual goal is is build TensorFlow from source using bazel.

Thank you!

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885.

janakdr picture janakdr on 12 Feb 2016

So I added the line: build --jobs=10 in a file mybazelrc. Then I performed:
export BAZELRC=/path/to/mybazelrc

Unfortunately after running ./compile.sh, I get the same error. I even tried build --jobs=1 but no luck.

ghost picture ghost on 12 Feb 2016

Can you put some nonsense in the file mybazelrc, to confirm that Bazel is
picking it up?

On Fri, Feb 12, 2016 at 2:28 PM, snazzman [email protected] wrote:

So I added the line: build --jobs=10 in a file mybazelrc. Then I
performed:
export BAZELRC=/path/to/mybazelrc

Unfortunately after running ./compile.sh, I get the same error. I even
tried build --jobs=1 but no luck.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183458134.

janakdr picture janakdr on 12 Feb 2016

It is definitely not picking it up. Just put 0xdeadbeef in there and it didn't complain :)

So when I do: vi $BAZELRC, the file does come up. But it is not been picked by bazel however.

ghost picture ghost on 12 Feb 2016

That reaches the limits of my compile.sh debugging. Other Bazel people?

On Fri, Feb 12, 2016 at 2:36 PM, snazzman [email protected] wrote:

It is definitely not picking it up. Just put deadbeef in there are it
didn't complain :)

So when I do: `vi $BAZELRC', the file does come up. But it is not been
picked by bazel however.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183460451.

janakdr picture janakdr on 12 Feb 2016

Try to export BAZEL_EXTRA_ARGS='--jobs 4'

The new bootstrap cannot parse bazelrc.

On Fri, Feb 12, 2016, 8:40 PM Janak Ramakrishnan [email protected]
wrote:

That reaches the limits of my compile.sh debugging. Other Bazel people?

On Fri, Feb 12, 2016 at 2:36 PM, snazzman [email protected]
wrote:

It is definitely not picking it up. Just put deadbeef in there are it
didn't complain :)

So when I do: `vi $BAZELRC', the file does come up. But it is not been
picked by bazel however.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183460451.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183461630.

damienmg picture damienmg on 12 Feb 2016

Nope, still the same error:

$ export BAZEL_EXTRA_ARGS='--jobs 2'                                                                                                                                                                                                         
$ ./compile.sh                                                                                                                                                                                                                               
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
.INFO: Elapsed time: 0.734s
unable to create new native thread

Blaze ran out of memory and crashed.

Building output/bazel
ghost picture ghost on 12 Feb 2016

oops my bad it is EXTRA_BAZEL_ARGS not BAZEL_EXTRA_ARGS

On Fri, Feb 12, 2016 at 8:48 PM snazzman [email protected] wrote:

Nope, still the same error:

$ export BAZEL_EXTRA_ARGS='--jobs 2'
$ ./compile.sh
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO: ./compile.sh compile /path/to/bazel
🍃 Building Bazel from scratch......
🍃 Building Bazel with Bazel.
.INFO: Elapsed time: 0.734s
unable to create new native thread

Blaze ran out of memory and crashed.

Building output/bazel

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183464390.

damienmg picture damienmg on 12 Feb 2016

Still no luck:

$ export EXTRA_BAZEL_ARGS='--jobs 1'                                         
$ ./compile.sh                                                               
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
.INFO: Elapsed time: 0.686s
unable to create new native thread

Blaze ran out of memory and crashed.

Building output/bazel

Is there a way to dump out more information or to look at a log somewhere to figure out more details about this crash? Also, running ulimit -a on the system gives the following:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030968
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 250
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
ghost picture ghost on 12 Feb 2016

Sorry, I should've checked more carefully before posting the previous message. There are log files under the main directory. I'll take a look at them and try to figure out what is going on and will reach back if I still need help.

ghost picture ghost on 12 Feb 2016

You can check $TMP/bazel.INFO to see the actual invocation of Bazel. See if
your --jobs flag showed up there.

On Fri, Feb 12, 2016 at 4:59 PM, snazzman [email protected] wrote:

Sorry, I should've check more carefully before posting the previous
message. There are log files under the main directory. I'll take a look at
them and try to figure out what is going on and will reach back if I still
need help.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183503668.

janakdr picture janakdr on 12 Feb 2016

FYI
scripts/bootstrap/compile.sh line 216
and
scripts/bootstrap/bootstrap.sh line 43

Are the places this call of bazel is done. It's done outside of the C++
client to avoid troubles with C++ compilation on various platforms.

On Fri, Feb 12, 2016 at 11:33 PM Janak Ramakrishnan <
[email protected]> wrote:

You can check $TMP/bazel.INFO to see the actual invocation of Bazel. See if
your --jobs flag showed up there.

On Fri, Feb 12, 2016 at 4:59 PM, snazzman [email protected]
wrote:

Sorry, I should've check more carefully before posting the previous
message. There are log files under the main directory. I'll take a look
at
them and try to figure out what is going on and will reach back if I
still
need help.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183503668.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183512603.

damienmg picture damienmg on 12 Feb 2016

erf and OUTPUT_DIR is a tempdir removed on exit :/ so you probably have no
log. You could comment out
scripts/bootstrap/buildenv.sh line 49
to
not delete the directory upon exit.

On Fri, Feb 12, 2016 at 11:35 PM Damien Martin-guillerez <
[email protected]> wrote:

FYI
scripts/bootstrap/compile.sh line 216
and
scripts/bootstrap/bootstrap.sh line 43

Are the places this call of bazel is done. It's done outside of the C++
client to avoid troubles with C++ compilation on various platforms.

On Fri, Feb 12, 2016 at 11:33 PM Janak Ramakrishnan <
[email protected]> wrote:

You can check $TMP/bazel.INFO to see the actual invocation of Bazel. See
if
your --jobs flag showed up there.

On Fri, Feb 12, 2016 at 4:59 PM, snazzman [email protected]
wrote:

Sorry, I should've check more carefully before posting the previous
message. There are log files under the main directory. I'll take a look
at
them and try to figure out what is going on and will reach back if I
still
need help.

—
Reply to this email directly or view it on GitHub
<https://github.com/bazelbuild/bazel/issues/885#issuecomment-183503668
.

—
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/885#issuecomment-183512603.

damienmg picture damienmg on 12 Feb 2016

So I solved the 'out of memory' issue and the build proceeds forward. Unfortunately it still fails:
ERROR: /home/scratch/Simulator/TensorFlow/bazel/third_party/ijar/BUILD:16:1: C++ compilation of rule '//third_party/ijar:zip' failed: gcc failed: error executing command

It is clear what the problem is because further down it does:

  /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel
_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/third_party/ijar/_objs/zip/third_party/ijar/zip.pic.o' -M
D -MF bazel-out/local_linux-fastbuild/bin/third_party/ijar/_objs/zip/third_party/ijar/zip.pic.d -fPIC -c third_party/ijar/zip.cc -o bazel-out/local_linux-fastbuild/bin/third_party/ijar/_objs/zip/third_party/ijar/zip.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: gcc f
ailed: error executing command

It is picking up GCC from the default location which is pretty old instead of the PATH variable. My understanding is that we have to modify the tools/cpp/CROSSTOOL file and specify the paths there. But this is a long file. Can you give me some hints as to what locations I need to update?

ghost picture ghost on 13 Feb 2016

Look around here:
https://github.com/bazelbuild/bazel/blob/master/tools/cpp/CROSSTOOL#L91

damienmg picture damienmg on 13 Feb 2016

So using the suggestions given here: https://github.com/bazelbuild/bazel/issues/760, here is how I modified tools/cpp/CROSSTOOL

   86                                                                                                                             
!  87   tool_path { name: "ar" path: "/home/utils/binutils-2.24/bin/ar" }                                                         
!  88   tool_path { name: "compat-ld" path: "/home/utils/binutils-2.24/bin/ld" }                                                  
!  89   tool_path { name: "cpp" path: "/home/utils/gcc-4.8.4/bin/cpp" }                                                           
   90   tool_path { name: "dwp" path: "/usr/bin/dwp" }                                                                            
!  91   tool_path { name: "gcc" path: "/home/utils/gcc-4.8.4/bin/gcc" }                                                           
   92   cxx_flag: "-std=c++0x"                                                                                                    
!  93   linker_flag: "-lstdc++ -L/home/utils/gcc-4.8.4/lib -L/home/utils/gcc-4.8.4/lib64"                                         
   94   linker_flag: "-B/usr/bin/"                                                                                                
+  95   linker_flag: "-Wl,-R/home/utils/gcc-4.8.4/lib64"                                                                          
   96                                                                                                                             
   97   # TODO(bazel-team): In theory, the path here ought to exactly match the path                                              
   98   # used by gcc. That works because bazel currently doesn't track files at                                                  
   99   # absolute locations and has no remote execution, yet. However, this will need                                            
  100   # to be fixed, maybe with auto-detection?                                                                                 
+ 101   cxx_builtin_include_directory: "/home/utils/gcc-4.8.4/include"                                                            
+ 102   cxx_builtin_include_directory: "/home/utils/gcc-4.8.4/include/c++/4.8.4"                                                  
+ 103   cxx_builtin_include_directory: "/home/utils/gcc-4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include"                     
+ 104   cxx_builtin_include_directory: "/home/utils/gcc-4.8.4/lib/gcc/x86_64-unknown-linux-gnu/4.8.4/include-fixed"               
  105   cxx_builtin_include_directory: "/usr/lib/gcc/"                                                                            
  106   cxx_builtin_include_directory: "/usr/local/include"                                                                       
  107   cxx_builtin_include_directory: "/usr/include"                                                                             
! 108   tool_path { name: "gcov" path: "/home/utils/gcc-4.8.4/bin/gcov" }                                                         
  109                                                                                                                             
  110   # C(++) compiles invoke the compiler (as that is the one knowing where                                                    
  111   # to find libraries), but we provide LD so other rules can invoke the linker.                                             
! 112   tool_path { name: "ld" path: "/home/utils/binutils-2.24/bin/ld" }                                                         
  113                                                                                                                             
! 114   tool_path { name: "nm" path: "/home/utils/binutils-2.24/bin/nm" }                                                         
! 115   tool_path { name: "objcopy" path: "/home/utils/binutils-2.24/bin/objcopy" }                                               
  116   objcopy_embed_flag: "-I"                                                                                                  
  117   objcopy_embed_flag: "binary"                                                                                              
! 118   tool_path { name: "objdump" path: "/home/utils/binutils-2.24/bin/objdump" }                                               
! 119   tool_path { name: "strip" path: "/home/utils/binutils-2.24/bin/strip" }                                                   
  120

Now, I get the following error:

ERROR: /home/scratch/Simulator/TensorFlow/bazel/src/main/tools/BUILD:31:1: C++ compilation of rule '//src/main/tools:namespace-sandbox' failed: gcc failed: error executing command

/home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.d -fPIC -c src/main/tools/namespace-sandbox.c -o bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: gcc failed: error executing command

/home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.d -fPIC -c src/main/tools/namespace-sandbox.c -o bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
INFO: Elapsed time: 4.390s, Critical Path: 0.82s

Building output/bazel

Any suggestions?

For full disclosure, here are the set of commands I'm running:

GCC_VER=/home/utils/gcc-4.8.4
export PATH=$GCC_VER/bin:$PATH 
export LD_LIBRARY_PATH=$GCC_VER/lib64:$LD_LIBRARY_PATH     
export JAVA_HOME=/home/utils/java/jdk1.8.0_51/
export EXTRA_BAZEL_ARGS='--jobs 4'  
./compile.sh
ghost picture ghost on 13 Feb 2016

We are missing some part of the log, right before ERROR you usually have the compiler error.

damienmg picture damienmg on 13 Feb 2016

Sorry for the delay. Here is the complete log:

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
INFO: Found 1 target...
ERROR: /home/scratch/Simulator/TensorFlow/bazel/src/main/tools/BUILD:31:1: C++ compilation of rule '//src/main/tools:namespace-sandbox' failed: gcc failed: error executing command
  (cd /tmp/bazel.2KFlkP3o/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.d -fPIC -c src/main/tools/namespace-sandbox.c -o bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: gcc failed: error executing command
  (cd /tmp/bazel.2KFlkP3o/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.d -fPIC -c src/main/tools/namespace-sandbox.c -o bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/namespace-sandbox/src/main/tools/namespace-sandbox.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
INFO: Elapsed time: 6.279s, Critical Path: 1.22s

Building output/bazel
ghost picture ghost on 16 Feb 2016

oh... you can replace the block at https://github.com/bazelbuild/bazel/blob/master/src/main/tools/BUILD#L33 with srcs = ["namespace-sandbox-dummy.c"], and the one at https://github.com/bazelbuild/bazel/blob/master/src/main/tools/BUILD#L42 with deps = [],

which gives you the full rule:

cc_binary(
    name = "namespace-sandbox",
    srcs = ["namespace-sandbox-dummy.c"],
    copts = ["-std=c99"],
    linkopts = ["-lm"],
    deps = [],
)

to replace line 31-52

damienmg picture damienmg on 17 Feb 2016

I tried the above suggestion. Now I get this error:

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
INFO: Found 1 target...
ERROR: /home/scratch/Simulator/TensorFlow/bazel/src/main/native/BUILD:30:1: C++ compilation of rule '//src/main/native:libunix.so' failed: gcc failed: error executing command
  (cd /tmp/bazel.OJuRnnNv/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -DBLAZE_OPENSOURCE -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem src/main/native -isystem bazel-out/local_linux-fastbuild/genfiles/src/main/native -isystem external/bazel_tools/tools/cpp/gcc3 -fPIC '-DBLAZE_JAVA_CPU=k8' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.d -fPIC -c src/main/native/unix_jni.cc -o bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: gcc failed: error executing command
  (cd /tmp/bazel.OJuRnnNv/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -DBLAZE_OPENSOURCE -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem src/main/native -isystem bazel-out/local_linux-fastbuild/genfiles/src/main/native -isystem external/bazel_tools/tools/cpp/gcc3 -fPIC '-DBLAZE_JAVA_CPU=k8' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.d -fPIC -c src/main/native/unix_jni.cc -o bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
INFO: Elapsed time: 6.736s, Critical Path: 1.80s

Building output/bazel

I also tried setting needsPic: false in tools/cpp/CROSSTOOL. In this case I get the following error:

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
INFO: Found 1 target...
ERROR: /home/scratch/Simulator/TensorFlow/bazel/src/main/native/BUILD:30:1: C++ compilation of rule '//src/main/native:libunix.so' failed: gcc failed: error executing command
  (cd /tmp/bazel.UxcQK9G8/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -DBLAZE_OPENSOURCE -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem src/main/native -isystem bazel-out/local_linux-fastbuild/genfiles/src/main/native -isystem external/bazel_tools/tools/cpp/gcc3 -fPIC '-DBLAZE_JAVA_CPU=k8' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.d -c src/main/native/unix_jni.cc -o bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: gcc failed: error executing command
  (cd /tmp/bazel.UxcQK9G8/out/bazel && \
  exec env - \
    PATH=/home/utils/gcc-4.8.4/bin:/home/lsf_linux/9.1/linux2.6-glibc2.3-x86_64/bin:/home/utils/Python-2.7.8/bin/:/home/utils/coreutils-8.23/bin:/home/utils/vim-7.4.712/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/home/utils/vim-7.4/bin:/home/utils/openssh-7.0p1/bin:/home/utils/llvm-3.6.0/bin:/home/utils/gdb-7.9/bin:/home/utils/valgrind-3.8.1/bin:/home/utils/ctags-5.8/bin:/home/utils/cmake-3.0.1/bin:/home/utils/perl-5.8.8/bin:/home/utils/textlive-2014/bin/x86_64-linux:/home/utils/ruby-2.1.3/bin:/home/utils/xclip-0.11/bin:/home/utils/scons-2.3.0/bin:/home/utils/tmux-1.9a/bin:/home/utils/Python-2.7.8/bin/:/home/utils/p4-2014.1:/home/utils/git-2.3.0/bin:/home/utils/mercurial-3.1.1/bin:/home/utils/bin:/home/nv/bin:/home/gnu/bin:/usr/local/lsf/bin:/usr/bin:/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin \
  /home/utils/gcc-4.8.4/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -DBLAZE_OPENSOURCE -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-fastbuild/genfiles/external/bazel_tools -isystem src/main/native -isystem bazel-out/local_linux-fastbuild/genfiles/src/main/native -isystem external/bazel_tools/tools/cpp/gcc3 -fPIC '-DBLAZE_JAVA_CPU=k8' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.o' -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.d -c src/main/native/unix_jni.cc -o bazel-out/local_linux-fastbuild/bin/src/main/native/_objs/libunix.so/src/main/native/unix_jni.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
INFO: Elapsed time: 6.689s, Critical Path: 1.43s

Building output/bazel
ghost picture ghost on 17 Feb 2016

what happens when you execute the command outside of bazel?

damienmg picture damienmg on 19 Feb 2016

@snazzman how did you solve the memory issue finally? I do
~$ export EXTRA_BAZEL_ARGS='--jobs 1'
~$ ./compile.sh
and it still gives me:
unable to create new native thread

Blaze ran out of memory and crashed.

Solved by: ulimit -u 100000 (in bash shell)

po0ya picture po0ya on 24 Feb 2016
❤1 👍1

I've abandoned trying to get this to work. Closing ticket.

ghost picture ghost on 8 Mar 2016

It worked for me by setting --jobs 10 and --ram_utilization_factor 25.
Basically i am trying to build the tensorflow python pip package from source, so the command now becomes,

bazel build --jobs 10 --ram_utilization_factor 25 -c opt //tensorflow/tools/pip_package:build_pip_package

I think this is happening cause of the poor specs of the machine.

sushantMoon picture sushantMoon on 5 Jul 2016
❤1 👍1

Thanks very much @po0ya ,I solved the problem by setting ulimit -u 100000 (in bash shell), how wonderful it is and it must be beyond my imagination!!!

ziyixi picture ziyixi on 19 May 2017

I am so upset about this threads.
ghost asked the question about Building Bazel tool from source and had out of memory issue. Lots of people answered the solution to fix out of memory when using Bazel --- not Building Bazel.

gittripley picture gittripley on 24 Oct 2019
😕1

How exactly does any of the above discussion this answer the original question - out of heap memory for a bootstrap build of bazel?

dtsmith2001 picture dtsmith2001 on 9 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

PanJinquan picture PanJinquan  ·  3Comments
ajaysaini-sgvu picture ajaysaini-sgvu  ·  3Comments
meteorcloudy picture meteorcloudy  ·  3Comments
lisaonduty picture lisaonduty  ·  3Comments
ob picture ob  ·  3Comments