Bazel: -static-libstdc++ -static-libgcc does not work with MOSTLY STATIC linking

[htuch]

When the -static-libstdc++ -static-libgcc flags are supplied as linkopts to cc_binary and linkstatic = 1, the resulting binary is still dynamically linked against libgcc/libc++, witness:

(cd /usr/local/google/home/htuch/.cache/bazel/_bazel_htuch/89676793239ac96d94294e6c7a44597f/execroot/envoy && \
  exec env - \
  /google/src/files/151979262/depot/google3/third_party/crosstool/v18/stable/wrappers/x86_64-grtev4-linux-gnu/bin/gcc -o bazel-out/local-fastbuild/bin/source/exe/envoy-static -pthread '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -static-libstdc++ -static-libgcc '-fuse-ld=gold' -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/g
oogle/src/files/151979262/depot/google3/third_party/crosstool/v18/stable/wrappers/x86_64-grtev4-linux-gnu/bin -B/usr/bin '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -pass-exit-codes -Wl,-S -Wl,@bazel-out/local-fastbuild/bin/source/exe/envoy-static-2.params)
Target //source/exe:envoy-static up-to-date:
  bazel-bin/source/exe/envoy-static
INFO: Elapsed time: 168.321s, Critical Path: 50.97s
<static-link> htuch@htuch00:~/src/envoy ⊧ ldd bazel-bin/source/exe/envoy-static
        linux-vdso.so.1 =>  (0x00007ffc0c3c7000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f60e820f000)
        libstdc++.so.6 => /google/src/files/151979262/depot/google3/third_party/crosstool/v18/stable/toolchain/x86_64-grtev4-linux-gnu/lib64/libstdc++.so.6 (0x00007f60e80f0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f60e7dea000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f60e7bcc000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f60e7804000)
        /usr/grte/v4/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x000055f7c0973000)
        libgcc_s.so.1 => /google/src/files/151979262/depot/google3/third_party/crosstool/v18/stable/toolchain/x86_64-grtev4-linux-gnu/lib64/libgcc_s.so.1 (0x00007f60e75ed000)

From the compiler man page, it looks like these flags are only supposed to work under the g++ driver. Even manually rerunning the command under g++ doesn't fix this. I had to modify bazel-out/local-fastbuild/bin/source/exe/envoy-static-2.params to remove the -lstdc++.

The Envoy (https://github.com/lyft/envoy/) Bazel build needs this mostly static link, where the Envoy libraries, external deps and standard C/C++ libs are statically linked, but the rest of the system libs are dynamically linked.

Looks like this requires some additional Bazel implementation work to properly support (at least as far as CROSSTOOLS goes, maybe another kind of pseudo-static link type). Are there any suggested workarounds in the meanwhile?

CC: @mattklein123 https://github.com/lyft/envoy/issues/415

[aehlig]

over to our C++ expert, to triage further

[htuch]

FTR, our workaround was to fork cc_configure.bzl into our repo and make the modifications indicated in https://github.com/lyft/envoy/pull/782.

[jmillikin-stripe]

Here's a smaller reproduction than building all of Envoy:

#!/bin/bash
set -e

touch WORKSPACE
cat >hello.cc <<EOF
#include <iostream>

int main() {
  std::cout << "hello world!\n";
  return 0;
}
EOF
cat >BUILD <<EOF
cc_binary(
  name="hello",
  srcs=["hello.cc"],
  linkstatic=1,
  linkopts=[
    "-static-libstdc++",
    "-static-libgcc",
  ],
)
EOF

bazel build :hello
cp bazel-bin/hello hello-bazel

gcc -c hello.cc -o hello-gcc.o
gcc -o hello-gcc hello-gcc.o -static-libstdc++ -static-libgcc -lstdc++
g++ -o hello-g++ hello-gcc.o -static-libstdc++ -static-libgcc

ldd hello-{bazel,g++,gcc}

$ ./test.sh
.
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello
INFO: Elapsed time: 4.187s, Critical Path: 0.02s
hello-bazel:
    linux-vdso.so.1 =>  (0x00007ffe0bfda000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3fb5cb4000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3fb59ab000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3fb55e0000)
    /lib64/ld-linux-x86-64.so.2 (0x00005652fdb8e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3fb53ca000)
hello-g++:
    linux-vdso.so.1 =>  (0x00007fff9dadc000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f480aea6000)
    /lib64/ld-linux-x86-64.so.2 (0x000055c2a4dcf000)
hello-gcc:
    linux-vdso.so.1 =>  (0x00007ffe4f8f1000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe7ae568000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe7ae19e000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe7ade94000)
    /lib64/ld-linux-x86-64.so.2 (0x0000559b6e494000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe7adc7e000)

We can see that gcc -lstdc++ has different linker behavior than g++, and the presence of libstdc++.so.6 in the final ELF also drags in libm and libgcc_s via transitive dependencies.

Explicitly referencing libstdc++.a allows gcc to work as expected:

$ gcc -o hello-gcc-2 hello-gcc.o -static-libgcc -l:libstdc++.a
$ ldd hello-gcc-2
    linux-vdso.so.1 =>  (0x00007ffdc5fdf000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb4b0789000)
    /lib64/ld-linux-x86-64.so.2 (0x00005581b559b000)

[jmillikin-stripe]

Can we re-open this, since the fix got rolled back?

[kamalmarhubi]

What was the reason for the rollback, and is it something we can resolve soon?

[hlopko]

https://github.com/bazelbuild/bazel/issues/4474 was the main culprit. I don't think I will have time in Q2 to fix this.

[mzeren-vmw]

I opened PR https://github.com/bazelbuild/bazel/pull/6360 with a fix. However, not surprisingly, it breaks several CI tests. If anyone has time to help triage / fix the failures that would be greatly appreciated.

[hlopko]

The PR is merged, I'm closing this issue. Thank you Mark!