Rules_go: Cross compilation is broken with golang.org/x/sys/unix

[apesternikov]

I am trying to cross compile to linux binary on darwin_amd64 machine. Compiling to darwin_amd64 target works file, cross compiling to linux_amd64 produces the following error:

bash-3.2$ bazel build //:doit_cross
INFO: Analysed target //:doit_cross (0 packages loaded).
INFO: Found 1 target...
ERROR: /Users/pesterni/go/src/github.com/apesternikov/bazel_go_repr/vendor/golang.org/x/sys/unix/BUILD.bazel:3:1: GoCompile vendor/golang.org/x/sys/unix/linux_amd64_pure_stripped/go_default_library~/partial.a failed (Exit 1)
vendor/golang.org/x/sys/unix/flock.go:16:42: undefined: Flock_t
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:37:18: undefined: Cmsghdr
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:43:9: undefined: Cmsghdr
2018/01/04 13:56:51 error running compiler: exit status 1
Target //:doit_cross failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.878s, Critical Path: 0.16s

This problem makes it impossible to build a linux binary (and, consequentially, docker image) referring golang.org/x/sys/unix or any of 2244 packages dependent on it https://godoc.org/golang.org/x/sys/unix?importers. We got this while trying to use github.com/fsnotify/fsnotify.

How to reproduce:
clone github.com/apesternikov/bazel_go_repr
bazel build //:doit works fine, bazel build //:doit_cross fails.
https://github.com/apesternikov/bazel_go_repr/blob/master/BUILD

[apesternikov]

I forgot to mention that GOOS=linux GOARCH=amd64 go install github.com/apesternikov/bazel_go_repr works fine of course.

[willtrking]

Running into the same issue, with the same error message

INFO: Analysed target //apparatus/cmd:cmd (1 packages loaded).
INFO: Found 1 target...
ERROR: /Users/willtrking/GoDev/src/github.com/willtrking/mono-test/vendor/golang.org/x/sys/unix/BUILD.bazel:3:1: GoCompile vendor/golang.org/x/sys/unix/linux_amd64_pure_stripped/go_default_library~/partial.a failed (Exit 1)
vendor/golang.org/x/sys/unix/flock.go:16:42: undefined: Flock_t
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:37:18: undefined: Cmsghdr
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:43:9: undefined: Cmsghdr
2018/01/13 12:43:15 error running compiler: exit status 1
Target //apparatus/cmd:cmd failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.346s, Critical Path: 0.66s
FAILED: Build did NOT complete successfully

[ghost]

Running into the same issue, with the same error message

library~/partial.a failed (Exit 1)
vendor/golang.org/x/sys/unix/flock.go:16:42: undefined: Flock_t
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:37:18: undefined: Cmsghdr
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:43:9: undefined: Cmsghdr
2018/01/16 12:03:28 error running compiler: exit status 1
Target //:inframgr-linux-amd64 failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.734s, Critical Path: 0.26s
FAILED: Build did NOT complete successfully

[fanlw0816]

Running into the same issue, with the same error message

# golang.org/x/sys/unix
../../../../../golang.org/x/sys/unix/flock.go:18: undefined: Flock_t
../../../../../golang.org/x/sys/unix/sockcmsg_linux.go:14: undefined: Ucred
../../../../../golang.org/x/sys/unix/sockcmsg_linux.go:27: undefined: Ucred
../../../../../golang.org/x/sys/unix/sockcmsg_unix.go:42: undefined: Cmsghdr

[jayconrod]

Just to explain what's happening here, most of the sources in golang.org/x/sys/unix have build constraints. Gazelle (via go_repository) interprets these and generates sources lists that look like this:

go_library(
    name = "go_default_library",
    srcs = select({
        "@io_bazel_rules_go//go/platform:darwin": [
            "constants.go",
            "dev_darwin.go",
            "dirent.go",
            "env_unix.go",
            "flock.go",
            "pagesize_unix.go",
            "race0.go",
            "sockcmsg_unix.go",
            "str.go",
            "syscall.go",
            "syscall_bsd.go",
            "syscall_darwin.go",
            "syscall_unix.go",
            "syscall_unix_gc.go",
            "timestruct.go",
        ],
        "@io_bazel_rules_go//go/platform:linux": [
            "affinity_linux.go",
            "bluetooth_linux.go",
            "constants.go",
            "dev_linux.go",
            "dirent.go",
            "env_unix.go",
            "flock.go",
            "pagesize_unix.go",
            "race0.go",
            "sockcmsg_linux.go",
            "sockcmsg_unix.go",
            "str.go",
            "syscall.go",
            "syscall_linux.go",
            "syscall_linux_gc.go",
            "syscall_unix.go",
            "syscall_unix_gc.go",
            "timestruct.go",
        ],
        ...

An appropriate list of sources is selected by Bazel using the target platform, which is usually provided on the command line. However, when a go_binary is cross-compiled by setting the goos / goarch attributes, the Bazel target platform is not explicitly set, so these select expressions are evaluated for the default (host) platform. That means go_library will get the wrong sources. There's not really any way we can fix this, other than to recommend that goos / goarch attributes are never used with libraries that include select expression that depend on the platform. Sorry that's not helpful.

The other approach to cross-compiling is to set the target platform on the command line instead of in go_binary.

$ bazel build --platforms=@io_bazel_rules_go//go/toolchain:darwin_amd64 //:cmd

This seems to be broken for a different reason. I've opened #1304 to track that. We will fix that soon.

Unfortunately, setting the target platform on the command line doesn't work well for multi-platform builds, e.g., if you want to build a release package that includes binaries for multiple platforms. There's no good solution for this yet.

[apesternikov]

We found a workaround for this problem. Apparently this 'select' is not required at all. If you try to compile all sources for all platforms, underlying compiler is following // +build specs and produce correct result. Here is our vendor/golang.org/x/sys/unix/BUILD:

`load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
name = "go_default_library",
srcs = [
"asm_darwin_386.s",
"asm_darwin_amd64.s",
"asm_darwin_arm.s",
"asm_darwin_arm64.s",
"asm_dragonfly_amd64.s",
"asm_freebsd_386.s",
"asm_freebsd_amd64.s",
"asm_freebsd_arm.s",
"asm_linux_386.s",
"asm_linux_amd64.s",
"asm_linux_arm.s",
"asm_linux_arm64.s",
"asm_linux_mips64x.s",
"asm_linux_mipsx.s",
"asm_linux_ppc64x.s",
"asm_linux_s390x.s",
"asm_netbsd_386.s",
"asm_netbsd_amd64.s",
"asm_netbsd_arm.s",
"asm_openbsd_386.s",
"asm_openbsd_amd64.s",
"asm_openbsd_arm.s",
"asm_solaris_amd64.s",
"bluetooth_linux.go",
"cap_freebsd.go",
"constants.go",
"dev_darwin.go",
"dev_dragonfly.go",
"dev_freebsd.go",
"dev_linux.go",
"dev_netbsd.go",
"dev_openbsd.go",
"dirent.go",
"endian_big.go",
"endian_little.go",
"env_unix.go",
"env_unset.go",
"errors_freebsd_386.go",
"errors_freebsd_amd64.go",
"errors_freebsd_arm.go",
"flock.go",
"flock_linux_32bit.go",
"gccgo.go",
"gccgo_linux_amd64.go",
"mkpost.go",
"openbsd_pledge.go",
"pagesize_unix.go",
"race.go",
"race0.go",
"sockcmsg_linux.go",
"sockcmsg_unix.go",
"str.go",
"syscall.go",
"syscall_bsd.go",
"syscall_darwin.go",
"syscall_darwin_386.go",
"syscall_darwin_amd64.go",
"syscall_darwin_arm.go",
"syscall_darwin_arm64.go",
"syscall_dragonfly.go",
"syscall_dragonfly_amd64.go",
"syscall_freebsd.go",
"syscall_freebsd_386.go",
"syscall_freebsd_amd64.go",
"syscall_freebsd_arm.go",
"syscall_linux.go",
"syscall_linux_386.go",
"syscall_linux_amd64.go",
"syscall_linux_amd64_gc.go",
"syscall_linux_arm.go",
"syscall_linux_arm64.go",
"syscall_linux_mips64x.go",
"syscall_linux_mipsx.go",
"syscall_linux_ppc64x.go",
"syscall_linux_s390x.go",
"syscall_linux_sparc64.go",
"syscall_netbsd.go",
"syscall_netbsd_386.go",
"syscall_netbsd_amd64.go",
"syscall_netbsd_arm.go",
"syscall_no_getwd.go",
"syscall_openbsd.go",
"syscall_openbsd_386.go",
"syscall_openbsd_amd64.go",
"syscall_openbsd_arm.go",
"syscall_solaris.go",
"syscall_solaris_amd64.go",
"syscall_unix.go",
"syscall_unix_gc.go",
"timestruct.go",
"types_darwin.go",
"types_dragonfly.go",
"types_freebsd.go",
"types_netbsd.go",
"types_openbsd.go",
"types_solaris.go",
"zerrors_darwin_386.go",
"zerrors_darwin_amd64.go",
"zerrors_darwin_arm.go",
"zerrors_darwin_arm64.go",
"zerrors_dragonfly_amd64.go",
"zerrors_freebsd_386.go",
"zerrors_freebsd_amd64.go",
"zerrors_freebsd_arm.go",
"zerrors_linux_386.go",
"zerrors_linux_amd64.go",
"zerrors_linux_arm.go",
"zerrors_linux_arm64.go",
"zerrors_linux_mips.go",
"zerrors_linux_mips64.go",
"zerrors_linux_mips64le.go",
"zerrors_linux_mipsle.go",
"zerrors_linux_ppc64.go",
"zerrors_linux_ppc64le.go",
"zerrors_linux_s390x.go",
"zerrors_linux_sparc64.go",
"zerrors_netbsd_386.go",
"zerrors_netbsd_amd64.go",
"zerrors_netbsd_arm.go",
"zerrors_openbsd_386.go",
"zerrors_openbsd_amd64.go",
"zerrors_openbsd_arm.go",
"zerrors_solaris_amd64.go",
"zptrace386_linux.go",
"zptracearm_linux.go",
"zptracemips_linux.go",
"zptracemipsle_linux.go",
"zsyscall_darwin_386.go",
"zsyscall_darwin_amd64.go",
"zsyscall_darwin_arm.go",
"zsyscall_darwin_arm64.go",
"zsyscall_dragonfly_amd64.go",
"zsyscall_freebsd_386.go",
"zsyscall_freebsd_amd64.go",
"zsyscall_freebsd_arm.go",
"zsyscall_linux_386.go",
"zsyscall_linux_amd64.go",
"zsyscall_linux_arm.go",
"zsyscall_linux_arm64.go",
"zsyscall_linux_mips.go",
"zsyscall_linux_mips64.go",
"zsyscall_linux_mips64le.go",
"zsyscall_linux_mipsle.go",
"zsyscall_linux_ppc64.go",
"zsyscall_linux_ppc64le.go",
"zsyscall_linux_s390x.go",
"zsyscall_linux_sparc64.go",
"zsyscall_netbsd_386.go",
"zsyscall_netbsd_amd64.go",
"zsyscall_netbsd_arm.go",
"zsyscall_openbsd_386.go",
"zsyscall_openbsd_amd64.go",
"zsyscall_openbsd_arm.go",
"zsyscall_solaris_amd64.go",
"zsysctl_openbsd_386.go",
"zsysctl_openbsd_amd64.go",
"zsysctl_openbsd_arm.go",
"zsysnum_darwin_386.go",
"zsysnum_darwin_amd64.go",
"zsysnum_darwin_arm.go",
"zsysnum_darwin_arm64.go",
"zsysnum_dragonfly_amd64.go",
"zsysnum_freebsd_386.go",
"zsysnum_freebsd_amd64.go",
"zsysnum_freebsd_arm.go",
"zsysnum_linux_386.go",
"zsysnum_linux_amd64.go",
"zsysnum_linux_arm.go",
"zsysnum_linux_arm64.go",
"zsysnum_linux_mips.go",
"zsysnum_linux_mips64.go",
"zsysnum_linux_mips64le.go",
"zsysnum_linux_mipsle.go",
"zsysnum_linux_ppc64.go",
"zsysnum_linux_ppc64le.go",
"zsysnum_linux_s390x.go",
"zsysnum_linux_sparc64.go",
"zsysnum_netbsd_386.go",
"zsysnum_netbsd_amd64.go",
"zsysnum_netbsd_arm.go",
"zsysnum_openbsd_386.go",
"zsysnum_openbsd_amd64.go",
"zsysnum_openbsd_arm.go",
"zsysnum_solaris_amd64.go",
"ztypes_darwin_386.go",
"ztypes_darwin_amd64.go",
"ztypes_darwin_arm.go",
"ztypes_darwin_arm64.go",
"ztypes_dragonfly_amd64.go",
"ztypes_freebsd_386.go",
"ztypes_freebsd_amd64.go",
"ztypes_freebsd_arm.go",
"ztypes_linux_386.go",
"ztypes_linux_amd64.go",
"ztypes_linux_arm.go",
"ztypes_linux_arm64.go",
"ztypes_linux_mips.go",
"ztypes_linux_mips64.go",
"ztypes_linux_mips64le.go",
"ztypes_linux_mipsle.go",
"ztypes_linux_ppc64.go",
"ztypes_linux_ppc64le.go",
"ztypes_linux_s390x.go",
"ztypes_linux_sparc64.go",
"ztypes_netbsd_386.go",
"ztypes_netbsd_amd64.go",
"ztypes_netbsd_arm.go",
"ztypes_openbsd_386.go",
"ztypes_openbsd_amd64.go",
"ztypes_openbsd_arm.go",
"ztypes_solaris_amd64.go",
],
# cgo = True,
importpath = "golang.org/x/sys/unix",
visibility = ["//visibility:public"],
)
Also, you need to prevent this file from being regenerated so we have a line in vendor/BUILD: # gazelle:exclude golang.org/x/sys/unix`

[jayconrod]

Seems like a good short-term solution.

I think the only reason we use select expressions on srcs is a minor performance improvement: the compiler is passed fewer sources. Since our compiler wrapper checks build constraints anyway, it's safe to pass it more than necessary. Maybe Gazelle should stop generating select for srcs.

deps is another story. It's important that we don't attempt to compile a Windows-only dependency if it's selected out anyway. The build may fail, and it may cause us to fetch more repositories than necessary.

golang.org/x/sys/unix doesn't have any deps though, so at least that works in this case.

[steeve]

I second that, as it seems config_setting are not honored when building via the goos/goarch attrs on go_binary.

For instance, if I do:

select({
    "@io_bazel_rules_go//go/platform:linux": [
        "file.go",
    ],
})

go_binary(
    ...
    goos = "linux",
)

Then it seems with the current rules_go it won't match, unless built via the --platform command line argument.

EDIT: it seems you already commented that @jayconrod, sorry

[srikumarb]

Any better solution other than manually changing the BUILD files and removing select?

[jayconrod]

@srikumarb At the moment, no.

I'm planning to have Gazelle (and go_repository) stop generating these select expressions in srcs. I've filed bazelbuild/bazel-gazelle#134 to track this.

EDIT: Using --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 on the command line instead of setting goos and goarch on go_binary will give you the fewest issues if you are able to use it.

[srikumar-b]

@jayconrod, I tried specifying --platforms in build command but that throws lot of random import errors all the time:

exec env - \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/compile -go external/go_sdk/bin/go -root_file 'bazel-out/darwin-fastbuild/bin/external/io_bazel_rules_go/darwin_amd64_stripped/stdlib~/ROOT' -goos linux -goarch amd64 '-cgo=0' -
compiler_path /usr/bin -cc external/local_config_cc/cc_wrapper.sh -cpp_flag '-D_FORTIFY_SOURCE=1' -cpp_flag -fstack-protector -cpp_flag -Wthread-safety -cpp_flag -Wself-assign -cpp_flag -fno-omit-frame-pointer -cpp_flag -O0 -cpp_flag -DDEBUG -ld_flag -headerpad_max
_install_names -ld_flag -lc++ -ld_flag -no-canonical-prefixes -package_list external/go_sdk/packages.txt -src external/com_github_spf13_afero/mem/dir.go -src external/com_github_spf13_afero/mem/dirmap.go -src external/com_github_spf13_afero/mem/file.go -o 'bazel-ou
t/darwin-fastbuild/bin/external/com_github_spf13_afero/mem/linux_amd64_pure_stripped/go_default_library~/github.com/spf13/afero/mem.a' -trimpath . -I . -- -p github.com/spf13/afero/mem)

Use --sandbox_debug to see verbose messages from the sandbox
external/com_github_spf13_afero/mem/dirmap.go:16:8: can't find import: "sort"
2018/02/20 17:04:58 error running compiler: exit status 1

Do you have any idea why it does not work?

[srikumar-b]

@jayconrod, we have been using --cpu=k8 until 0.9.0 but its broken with these random import issues now from 0.10.0 and tried goos + pure and it fails with the select clause.

Any workaround is highly appreciated. we have many packages imported hence manually fixing select is not an option for me.

[jayconrod]

The import errors you mentioned sound like #1304, which was fixed in #1310. Do you still see that error on master? I'm planning to tag a new release tomorrow that will include that fix if you want to wait for that.

[srikumar-b]

@jayconrod could you let me know the path to use for master? I can quickly try it and confirm before you tag it.

[jayconrod]

@srikumar-b Sure, just replace the io_bazel_rules_go repository rule in WORKSPACE with this:

git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    commit = "4b5e4d361f3a54286260cc36c6371468d62662aa",
)

[srikumar-b]

@jayconrod works like a charm. Thanks a lot.
Kindly keep us posted when it is tagged.

[srikumar-b]

@jayconrod If you are going to tag it in next couple of hours, I would choose to wait and use the release tag directly. Thanks again.

[jayconrod]

Release is tagged now.

[srikumar-b]

@jayconrod Thanks Jay. I am getting an interesting error with the new release now...

I get build errors with dependency resolution with package https://github.com/PuerkitoBio/purell because PuerkitoBio has caps and generated bazel files are in small case.

Build Error:

/private/var/tmp/_bazel_abcd/1b37658f9dfc35d8f935bda6ce8aa1f8/external/com_github_go_openapi_jsonreference/BUILD.bazel:3:1: no such package '@com_github_puerkitobio_purell//': The repository could not be resolved and referenced by '@com_github_go_openapi_jsonreference//:go_default_library'

Generated BUILD.bazel with 0.9.0:

go_library(
    name = "go_default_library",
    srcs = ["reference.go"],
    importpath = "github.com/go-openapi/jsonreference",
    visibility = ["//visibility:public"],
    deps = [
        "@com_github_PuerkitoBio_purell//:go_default_library",
        "@com_github_go_openapi_jsonpointer//:go_default_library",
    ],
)

Generated BUILD.bazel with 0.10.0:

go_library(
    name = "go_default_library",
    srcs = ["reference.go"],
    importpath = "github.com/go-openapi/jsonreference",
    visibility = ["//visibility:public"],
    deps = [
        "@com_github_go_openapi_jsonpointer//:go_default_library",
        "@com_github_puerkitobio_purell//:go_default_library",
    ],
)

[jayconrod]

@srikumar-b I've filed bazelbuild/bazel-gazelle#137 to address this.

In the mean time, you should be able to work around this by changing the name of the rule in WORKSPACE that imports this repository to com_github_puerkitobio_purell, which is what Gazelle blindly expects.

[srikumar-b]

Thanks @jayconrod I only see these entries in Gopkg.lock but not in WORKSPACE. Is it possible to override these entries from WORKSPACE?

[[projects]]
  name = "github.com/PuerkitoBio/purell"
  packages = ["."]
  revision = "0bcb03f4b4d0a9428594752bd2a3b9aa0a9d4bd4"
  version = "v1.1.0"

[[projects]]
  branch = "master"
  name = "github.com/PuerkitoBio/urlesc"
  packages = ["."]
  revision = "de5bf2ad457846296e2031421a34e2568e304e35"

[jayconrod]

I'm guessing you're using dep_import or something similar to create repository rules from Gopkg.lock at build time. I don't think that tool provides any way to customize the generated repository rules.

I'd recommend using gazelle update-repos -from_file=Gopkg.lock to import go_repository rules into WORKSPACE instead. You'll be able to manually change the repo names after that.

[srikumar-b]

@jayconrod I have manually added those to WORKSPACE but now I am getting this issue: https://github.com/kubernetes/kubernetes/issues/50975
The workaround mentioned at https://github.com/kubernetes/kubernetes/issues/50975#issuecomment-337823910 is not working anymore :-(

[jayconrod]

@srikumar-b That seems like an issue in the implementation of go_genrule in Kubernetes. I'm not familiar with the implementation of that rule. Maybe follow up on that issue or the dup linked at the bottom? From the comments it sounds like a generated file was supposed to be checked in.

[jayconrod]

(Re-triaging old issues)

Closing as obsolete. This issue covered a lot of problems, most of which have been fixed.

The main problem, that select does not work with goos and goarch set on go_binary is still a problem. #2219 is now the tracking issue for that.