Rules_go: Cannot build cgo libraries on gcc-6 systems

[ivucica]

I've encountered the same issue reported in https://github.com/kubernetes/kubernetes/issues/39819.

Repro steps

I've checked out a Cgo project, apcera/gssapi@5fb4217df13b8e6878046fe1e5c10e560e1b86dc, into vendor/github.com/apcera/gssapi.

I've then used the following minimal rule:

# vendor/BUILD
# -*-Python-*-
load("@io_bazel_rules_go//go:def.bzl", "go_library", "cgo_library")

cgo_library(
    name = "github.com/apcera/gssapi_c",
    srcs =
    glob([
        #"github.com/apcera/gssapi/*.go",
        "github.com/apcera/gssapi/buffer.go",
    ], exclude=[
        "github.com/apcera/gssapi/*_test.go",
        "github.com/apcera/gssapi/doc.go"])
     + [
        "bleh.c"
    ],
    clinkopts = [
        #"--no-pie", "-Wl,-no-pie",
        "-ldl",
    ],
)

go_library(
    name = "github.com/apcera/gssapi",
    library = ":github.com/apcera/gssapi_c",
)

Error received when building //vendor:github.com/apcera/gssapi_c is /usr/bin/ld: -r and -pie may not be used together.

Trying to pass clinkopts = ["--no-pie", "-Wl,-no-pie"] sometimes doesn't help, and sometimes results in cannot find "-lgcc_s".

"bleh.c" may be needed and is an empty file. bazel clean --expunge may be required to reproduce -lgcc_s. Building without bleh.c and using *.go should result in the same output as long as --expunge was run between switching between GCC 4.9 and GCC 6.

Cause

Quoting @mikedanese in https://github.com/kubernetes/kubernetes/issues/39819#issuecomment-275474974:

this is caused by gcc 6 on debian which enable -pie by default (whether you pass -pie or not). This is different then gcc-5 behavior. To fix this we need to pass -no-pie explicitly or stop using -r.

Versions

• GCC: I've used the Debian stable GCC 4.9, and Debian testing GCC 6.2.1-5 (package version 6.2.1-1).
• Bazel: 0.4.3 release
• Go rules: 81a7f69370a4917aee98c0b804626824f3203ad5

[ixdy]

@damienmg @pmbethe09 any update here?

[damienmg]

Also cc'ing @alandonovan if his team has idea.

[jayconrod]

I'll look into this.

[jayconrod]

In _cgo_object_impl, it looks like we're passing -Wl,-r to the compiler when we build the cgo object. We're also excluding flags starting with -Wl. This rule is internal, part of the guts of cgo_library, and it doesn't look like it observes clinkopts anyway. So that would be why -Wl,-no-pie doesn't work.

Maybe we should add -Wl,-no-pie to the flags in that function. I'll spin up a Debian VM, try to reproduce this, then confirm that works as soon as I can.

[justinsb]

I can confirm that adding the proposed arguments (and then loading my fork with local_repository) does indeed "works on my machine":

+++ b/go/def.bzl
@@ -953,6 +953,7 @@ def _cgo_object_impl(ctx):
     arguments += ["-shared", "-Wl,-all_load"]
   else:
     arguments += ["-Wl,-whole-archive"]
+  arguments += ["-Wl,-no-pie"]

   lo = ctx.files.src[-1]
   arguments += [lo.path]

However, I don't know enough to know if that is safe elsewhere. Let me know if there's anything I can do to provide identifying information for (my) debian stretch machine.

[jayconrod]

Sorry for the long silence. I haven't forgotten about this one.

I reproduced this issue in a Debian VM, and verified that adding -Wl,-no-pie fixes the problem on that machine. Unfortunately, adding that option breaks builds on Ubuntu 16.04 with GCC 4.8.4, and probably other builds, too. GCC fails without printing an error message or a nonzero exit code; it just doesn't produce an output.

We may need to do some configuration in the repository rules to check which linker options are supported. I don't think we can have a single set of options that works everywhere.

Somewhat related issue: golang/go #13937.

[lukaszgryglicki]

Still an issue on Ubuntu 16.10:
Linux vubuntu 4.8.0-22-generic #24-Ubuntu SMP Sat Oct 8 09:15:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Default gcc (mean gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005):

lukasz@vubuntu:~/dev/kubernetes/test-infra$ bazel build //...
Extracting Bazel installation...
...........
INFO: Found 391 targets...
ERROR: /home/lukasz/dev/kubernetes/test-infra/vendor/BUILD:363:1: Linking of rule '//vendor:github.com/mattn/go-sqlite3_cgo.cgo.dir/gen/_cgo_.o' failed: Process exited with status 1 [sandboxed].
/usr/bin/ld.gold: error: cannot find -lz
collect2: error: ld returned 1 exit status
Use --strategy=CppLink=standalone to disable sandboxing for the failing actions.
INFO: Elapsed time: 83.061s, Critical Path: 46.60s

or

lukasz@vubuntu:~/dev/kubernetes/test-infra$ bazel build //... --strategy=CGoObject=standalone
WARNING: /home/lukasz/.cache/bazel/_bazel_lukasz/6e62a321665c5aca3b6df992903fb76b/external/nodejs_linux_amd64/WORKSPACE:1: Workspace name in /home/lukasz/.cache/bazel/_bazel_lukasz/6e62a321665c5aca3b6df992903fb76b/external/nodejs_linux_amd64/WORKSPACE (@org_pubref_rules_node_toolchain) does not match the name given in the repository's definition (@nodejs_linux_amd64); this will cause a build error in future versions.
INFO: Found 391 targets...
ERROR: /home/lukasz/dev/kubernetes/test-infra/vendor/BUILD:363:1: Linking vendor/github.com/mattn/go-sqlite3_cgo.cgo.dir/gen/_all.o failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers ... (remaining 14 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
/usr/bin/ld.gold: fatal error: -pie and -r are incompatible
collect2: error: ld returned 1 exit status
INFO: Elapsed time: 0.904s, Critical Path: 0.04s

And when gcc downgraded to 5:

lukasz@vubuntu:~/dev/kubernetes/test-infra$ gcc --version
gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

lukasz@vubuntu:~/dev/kubernetes/test-infra$ gcc
gcc           gcc-5         gcc-6         gcc-ar        gcc-ar-5      gcc-ar-6      gcc-nm        gcc-nm-5      gcc-nm-6      gcc-ranlib    gcc-ranlib-5  gcc-ranlib-6  
lukasz@vubuntu:~/dev/kubernetes/test-infra$ gcc-5
gcc-5: fatal error: no input files
compilation terminated.
lukasz@vubuntu:~/dev/kubernetes/test-infra$ update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 1
update-alternatives: using /usr/bin/gcc-5 to provide /usr/bin/gcc (gcc) in auto mode
update-alternatives: error: error creating symbolic link '/etc/alternatives/gcc.dpkg-tmp': Permission denied
lukasz@vubuntu:~/dev/kubernetes/test-infra$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 1
update-alternatives: using /usr/bin/gcc-5 to provide /usr/bin/gcc (gcc) in auto mode
lukasz@vubuntu:~/dev/kubernetes/test-infra$ gcc --version
gcc (Ubuntu 5.4.1-2ubuntu2) 5.4.1 20160929

Gives:

lukasz@vubuntu:~/dev/kubernetes/test-infra$ bazel build //...
WARNING: /home/lukasz/.cache/bazel/_bazel_lukasz/6e62a321665c5aca3b6df992903fb76b/external/nodejs_linux_amd64/WORKSPACE:1: Workspace name in /home/lukasz/.cache/bazel/_bazel_lukasz/6e62a321665c5aca3b6df992903fb76b/external/nodejs_linux_amd64/WORKSPACE (@org_pubref_rules_node_toolchain) does not match the name given in the repository's definition (@nodejs_linux_amd64); this will cause a build error in future versions.
INFO: Found 391 targets...
ERROR: /home/lukasz/dev/kubernetes/test-infra/vendor/BUILD:363:1: undeclared inclusion(s) in rule '//vendor:github.com/mattn/go-sqlite3_cgo.cgo.dir/gen/_cgo_lib':
this rule is missing dependency declarations for the following files included by 'vendor/github.com/mattn/go-sqlite3_cgo.cgo.dir/gen/error.cgo2.c':
  '/usr/lib/gcc/x86_64-linux-gnu/5/include/stddef.h'.
INFO: Elapsed time: 16.618s, Critical Path: 13.64s

So I see no way to build test-infra on newest Ubuntu ... :(

[yohcop]

After upgrading to Ubuntu 17.04, same issue. Using clinkopts = ["--no-pie", "-Wl,-no-pie"] didn't work though It's like they are not on the command line anyway:

ERROR: /d/code/cardiocoach/lmfit/BUILD:3:1: Linking lmfit/go_default_library.cgo.dir/gen/_all.o failed: gcc failed: error executing command 
  (exec env - \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-fuse-ld=gold' -B/usr/bin -B/usr/bin -pass-exit-codes -o bazel-out/local-fastbuild/bin/lmfit/go_default_library.cgo.dir/gen/_all.o -nostdlib -Wl,-r -Wl,-whole-archive bazel-out/local-fastbuild/bin/lmfit/go_default_library.cgo.dir/gen/lib_cgo_lib.lo)

[ghost]

I've just run across this issue while attempting the the Bazel build of Kubernetes. I can provide my Bazel WORKSPACE if needed, but it's a bit verbose at this point. Assuming that's not needed, on Ubuntu 17.04 using default build toolchain, the following is enough to repro this error.

$ bazel build @io_kubernetes//test/...

[mkmik]

I'm currently working around it with:

bazel build --client_env=CC=clang ...

[mkmik]

except it doesn't work with people having clang-4.0 due to:

_cgo/BUILD.bazel:45:1: undeclared inclusion(s) in rule '@packaging//vendor/github.com/mmikulicic/jsonnet_cgo:cgo_default_library.cgo_c_lib':
this rule is missing dependency declarations for the following files included by 'external/packaging/vendor/github.com/mmikulicic/jsonnet_cgo/cgo_default_library.cgo_postselect.dir/external/packaging/vendor/github.com/mmikulicic/jsonnet_cgo/jsonnet.cgo2.c':
  '/usr/lib/llvm-4.0/lib/clang/4.0.0/include/stddef.h'
  '/usr/lib/llvm-4.0/lib/clang/4.0.0/include/stdarg.h'
  '/usr/lib/llvm-4.0/lib/clang/4.0.0/include/__stddef_max_align_t.h'.

this is quite annoying since my team runs on variety of ubuntu platforms

[mkmik]

Possibly related https://phabricator-dot-kythe-repo.appspot.com/D1161

[haavardw]

I'm successfully working around this on ubuntu 17.04 by adding this to my .bazelrc:

build --linkopt="-no-pie"

[jayconrod]

An update: https://golang.org/cl/64793 removes the partial link step in cgo in "go build".

I plan to remove this from the Bazel rules as well. This should fix this issue, since we'll no longer link with -r at any point.