Rules_go: go_binary cross-compilation

Created on 20 Aug 2016  Â·  51Comments  Â·  Source: bazelbuild/rules_go

What's the correct way to build a go_binary for a linux platform on osx? For example, bazel build //my/server where ":server" is a go_binary rule works on my osx machine but fails with "can't find import: html/template" when invoked with --cpu armeabi_v7a. I was expecting a toolchain for k8 but the only available --cpu options are:

ERROR: No toolchain found for cpu 'k8'. Valid cpus are: [
  darwin,
  armeabi-v7a,
  x64_windows_msvc,
].
enhancement

Most helpful comment

Cross compilation is launched as of this Monday.
You need Bazel 0.5.4 and the master branch of rules_go.

bazel build --cpu k8

when compiling on Mac will build a linux binary. More to come.

All 51 comments

Here's the error while attempting to compile the most trivial HelloWorld program:

ERROR: /Users/pcj/foo/go/BUILD:6:1: error executing shell command: 'rm -rf bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir && mkdir -p bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir && mkdir -p bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir/g...' failed: bash failed: error executing command 
  (cd /private/var/tmp/_bazel_pcj/75329c14c71b627aaf1fecd401bf1588/execroot/foo && \
  exec env - \
    GOARCH=arm \
    GOOS=linux \
  /bin/bash -c 'rm -rf bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir \
&& mkdir -p bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir \
&& mkdir -p bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir/foo/go \
&& ln -s ../../../../../../../../bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.a bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir/foo/go/server.a \
&& export PATH=$PATH:/usr/bin \
&& export GOROOT=$(pwd)/external/golang_darwin_amd64/go/bin/.. \
&& cd bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server.dir \
&& ../../../../../external/golang_darwin_amd64/go/bin/go tool link -L . -o foo/go/server -s -extld /bin/false -extldflags '\''-Wl,-S -Wl,-rpath,$ORIGIN/../ -Lfoo/'\'' foo/go/server.a \
&& mv -f foo/go/server ../../../../../bazel-out/stub_armeabi-v7a-fastbuild/bin/go/server'):\
com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
warning: unable to find runtime.a
warning: unable to find math.a
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x60 pc=0x148037]

goroutine 1 [running]:
panic(0x236a60, 0x8204160d0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
cmd/link/internal/ld.pclntab()
    /usr/local/go/src/cmd/link/internal/ld/pcln.go:360 +0x10d7
cmd/link/internal/ld.Ldmain()
    /usr/local/go/src/cmd/link/internal/ld/pobj.go:241 +0x1f33
cmd/link/internal/arm.Main()
    /usr/local/go/src/cmd/link/internal/arm/obj.go:44 +0x19
main.main()
    /usr/local/go/src/cmd/link/main.go:29 +0x191
Target //go:server failed to build
INFO: Elapsed time: 0.109s, Critical Path: 0.04s

Copying @yugui's response from bazel-discuss for context.

On my osx machine, I'd like compile a go_binary target for inclusion into a docker_build rule for alpine linux.
Anyone know if the rules_go supports cross-compilation? If so, what is the correct way to invoke bazel?

No, it does not support. There are two problems as you have already found out by yourself.

  1. Bazel refuses it, like ERROR: No toolchain found for cpu 'k8'. Valid cpus are: [... IIUC, Bazel is trying to locate C++ toolchain for the platform when it validate -c flag even if we are trying to build go binary. I wonder if this is an expected behavior.
  2. GOROOT does not contain a prebuilt std library for the platform. Probably we can extend go_repositories rules to cross compile the std library, but I want to think a bit more about the solution.

How would one approach this problem? In other words, How to break this problem down into actionable steps? Where to start?

So bazel has the idea of --target_environment which is the platform you are
compiling for being different from the host environment (where you are
compiling).
Go does cross compilation via GOOS and GOARCH
http://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5

So I think the magic here would be if the Go rules
inspected target_environment or some other flag and used that to drive
the GOOS and GOARCH flags.

On Mon, Aug 22, 2016 at 6:16 PM, Paul Cody Johnston <
[email protected]> wrote:

Copying @yugui https://github.com/yugui's response from bazel-discuss
https://groups.google.com/d/msg/bazel-discuss/YC_Fwl_cQa0/1x1JPQKNBQAJ
for context.

On my osx machine, I'd like compile a go_binary target for inclusion into
a docker_build rule for alpine linux.

Anyone know if the rules_go supports cross-compilation? If so, what is the
correct way to invoke bazel?

No, it does not support. There are two problems as you have already found
out by yourself.

1.

Bazel refuses it, like ERROR: No toolchain found for cpu 'k8'. Valid
cpus are: [... IIUC, Bazel is trying to locate C++ toolchain for the
platform when it validate -c flag even if we are trying to build go binary.
I wonder if this is an expected behavior.
2.

GOROOT does not contain a prebuilt std library for the platform.
Probably we can extend go_repositories rules to cross compile the std
library, but I want to think a bit more about the solution.


How would one approach this problem? In other words, How to break this
problem down into actionable steps? Where to start?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/rules_go/issues/70#issuecomment-241568469,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALF-0oHt5o0y9TvQlwlajSVUP5CiRLvAks5qih_HgaJpZM4Jo9IC
.

From the example above, GOARCH=arm and GOOS=linux were already correctly set by rules_go, but it does look like the go link failed due to lack of cross-compiled standard library.

go install _should_ work to automagically cross-compile the std library and save it under (for example) /private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/external/golang_darwin_amd64/go/pkg.

Since this directory is writable in a bazel install, it already sidesteps the typical problem of the GOROOT not being writable (like /usr/local/go as mentioned in the article)

For example, this works:

$ export GOROOT=$(bazel info output_base)/external/golang_darwin_amd64/go
$ export GOPATH=$(bazel info workspace)/go
$ export GOBIN=$(bazel info bazel-bin)
$ env \
 GOOS=linux \
 GOARCH=arm \
 $GOROOT/bin/go install go/src/foo.go 

Which creates the appropriate $GOROOT/pkg/linux_arm std library *.a files on my darwin machine.

@yugui are you sure we really need to install a prebuilt std library? It seems all we need to do is translate whatever is happening in go install into your lower level set of commands, (such as go link...)

I have a prototype of this working. Before sending a pull-request, I'd like to gather feedback on the design:

Use Case

My use case is to build a lightweight minimal docker container carrying only a single statically compiled go binary. To do this, I'd like to provide a label to the docker_build.files attribute. This label will point to the cross-compiled go binary file. For example:

docker_build(
    name = "service",
    base = "@org_pubref_rules_docker//docker:scratch",
    files = [":linux_arm64/foo"]
)

The combination of statically linked single go binaries and deterministic docker images is IMHO a killer bazel feature. A few background articles:

New attribute "targets": attr.output_list()

In order to support this, I've added a targets attribute to the go_binary rule having type output_list. The targets list follows the convention GOOS_GOARCH/FILENAME. In the go_binary rule, the target output file name is parsed to determine the desired platform configuration and matched against https://golang.org/doc/install/source#environment for a permitted configuration. Foreach platform, a separate cross-compiled binary file is produced. Here's how the rule looks:

go_binary(
    name = "foo",
    srcs = ["foo.go"],
    targets = ["linux_arm64/foo"],
)

The implementation involves changing the go_binary_impl into a for-loop over each desired configuration and calling go_library_impl foreach one, passing in the desired environment of GOOS and GOARCH along with the ctx argument. The standard library is built on-demand in emit_go_compile_action with an additional go install std command to build the stdlib foreach required platform.

Additional GOOS_GOARCH output path element

It also adds a new path element to the output_tree. Following the convention of the go toolchain to create GOROOT/pkg/GOOS_GOARCH foreach cross-compiled runtime/stdlib, bazel-bin looks like:

bazel-bin/go
bazel-bin/go/darwin_amd64
bazel-bin/go/darwin_amd64/foo.a
bazel-bin/go/darwin_amd64/foo.a.dir/github.com/pubref/grpc_greetertimer/go/foo.go
bazel-bin/go/foo
bazel-bin/go/foo.a
bazel-bin/go/foo.dir/github.com/pubref/grpc_greetertimer/godarwin_amd64/foo.a
bazel-bin/go/linux_arm64/foo
bazel-bin/go/linux_arm64/foo.a
bazel-bin/go/linux_arm64/foo.a.dir/github.com/pubref/grpc_greetertimer/go/foo.go
bazel-bin/go/linux_arm64/foo.dir/github.com/pubref/grpc_greetertimer/golinux_arm64/foo.a

Where foo and foo.a still correspond to outputs for the default host configuration (the current rules_go behavior). There are some tweaks todo here, but you get the idea. I believe this is the right idea, but as a consequence it affects the majority of def.bzl.

Thoughts/comments @pmbethe09 @yugui @hanwen @damienmg @kchodorow? I've yet to explore the interaction with cgo binaries and cpp crosstool, etc.

One issue I'm facing is cross-compilation of dependencies. For example, if a go_binary has a dependency on glog, the glog go_library target has no way of knowing that it should be cross-compiled, i.e., there's no way propagate information up the dependency chain before compile actions are executed.

I'm thinking that aspects may be the solution here, yes? Perhaps building cross-compiled targets within an aspect implementation is the way to go. There are few practical examples of aspects.

This sounds reasonable, but I don't know much about aspects either. Maybe
we can get damienmg to comment?

On Wed, Sep 7, 2016 at 8:00 PM, Paul Cody Johnston <[email protected]

wrote:

One issue I'm facing is cross-compilation of dependencies. For example, if
a go_binary has a dependency on glog, the glog go_library target has no way
of knowing that it should be cross-compiled, i.e., there's no way propagate
information up the dependency chain before compile actions are executed.

I'm thinking that aspects may be the solution here, yes? Perhaps building
cross-compiled targets within an aspect implementation is the way to go.
There are few practical examples of aspects.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/rules_go/issues/70#issuecomment-245456027,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALF-0vd3Q-upEoltHAT4LT1LLlQgI4o5ks5qn1AlgaJpZM4Jo9IC
.

I've been researching the aspect angle and I think it is a good solution. Here's how it works (different from the proposal above):

There is a new rule xgo_binary used for cross-compilation (maybe it should be called xgo_object since it's not executable). xgo_binary has a 'deps' attribute like so:

    "deps": attr.label_list(
      providers = [
        "direct_deps",
        "go_library_object",
        "transitive_go_importmap",
        "transitive_go_library_object",
        "transitive_cgo_deps",
        "xgo_result",
      ],
      aspects = [xgo_aspect],
    ),

The providers of deps are the same as go_library and go_binary, with an extra declared provider xgo_result. This will eventually hold a struct containing the output of an xgo_aspect application.

The xgo_aspect (below) declares that it propagates across the deps attribute, meaning that bazel will create a shadow graph of nodes with that attribute. In this case, it starts at the deps of an xgo_binary rule. Therefore, foreach transitive go_library or go_binary dependency of xgo_binary, apply xgo_aspect.

xgo_aspect = aspect(
  implementation = xgo_aspect_impl,
  attr_aspects = ["deps"],
)

The aspect implementation inspects ctx.rule.kind to see whether it is processing a go_library or go_binary dependency. It then performs the same emit_go_compile and emit_go_link actions as the normal go_library or go_binary rule would, effectively creating a shadow graph of cross-compiled targets.

def xgo_aspect_impl(target, ctx):
  kind = ctx.rule.kind
  outputs = []

  if kind == 'go_binary':
    outputs += _xgo_go_binary(target, ctx)
  elif kind == 'go_library':
    outputs += _xgo_go_library(target, ctx)
  else:
    fail("Unexpected dependency kind %s" % kind)

  return mk_xgo_result(outputs)

By the time xgo_binary_impl is processed, everything has been built, and it just needs to pluck out the completed cross-compiled binary object from the xgo_result provider and return that in it's output struct.

In order to not duplicate code however, it means re-engineering go_library_impl and go_binary_impl to create common entrypoints for either normal rules or aspect rules. Since aspect rules and normal rules have a different ctx object, this re-engineering involves making the implementations agnostic to the layout of the ctx argument. This is the part I'm working on now.

It should not change the semantics of go_library or go_binary, but it makes the implementation look a lot different. I'm a little worried that such a PR will be looked down upon however as it involves alot of change in one go. Unfortunately, I don't know how to do it incrementally either. Hopefully you @pmbethe09 and @yugui can take that into account. Certainly welcome input from @damienmg as well.

First I want to know what it the proper way of supporting multiple target platform in Bazel.
It looks to be --cpu, but it requires support in CROSSTOOL as @pcj pointed out.

In my understanding, it is hard to propagate platform information just because we are trying to workaround --cpu.

In my case I'd like to avoid use of the --cpu flag as I want to explicitly define the target platform in the rule. I want to use the output of the rule like a filegroup, to be used in the docker_build.files attribute.

Propogating the platform information turned out to require "parameterized aspects" discussed in https://bazel.io/designs/skylark/parameterized-aspects.html.

It's working well, I can now cross-compile static grpc-go apps and place them in minimal alpine-linux containers. I hope to push a PR today so you can take a look at it.

Cross-compiling assembly and cgo dependencies is still unsolved though, and I'd like to understand how better to incorporate it.

It seems we need a repo or set of repo(s) that contain viable crosstool setups for various architectures. Something like bazelbuild/rules_crosstool with a separate branch foreach platform would be great.

has there been any progress on this recently? we'd really like to use bazel to build kubernetes, but lack of cross-build support is hurting us.

Hi @ixdy I had this pretty much fully-implemented this on a separate branch in #98, but it was a fairly radical change that was too much to review in one go, so it got tabled.

I put forth a design document to detail the steps to move forward, but I could not really get consensus from the bazel team about if they liked the approach. There was lingering ambiguity on whether and/or how much rules_go should interact with the CROSSTOOL subsystem, etc. (my personal opinion: don't interact with it at all, because it may be changing significantly anyway over the next year).

Anyway, bottom line nobody seemed to care enough about it, or didn't feel they have enough perspective to take a stand. If there is renewed support, I'd take it up again.

Obviously kubernetes is a complex project... How much native c / c++ code is there along with the pure go codebase?

Kubernetes itself is almost no C, but we do have some cgo dependencies, mostly in vendored code, I think.

cc @mikedanese

Just to give a bit more context on Kubernetes' needs: we cross-compile to linux/{386,amd64,arm,arm64,ppcle64,s380x}, windows/{386,amd64}, and darwin/{386,amd64}, though not every binary targets every platform.

It'd be nice to be able to easily target just one arch (e.g. linux/amd64, if I'm running everything on linux), a subset (just linux/amd64 and darwin/amd64, if I'm building on a Mac), or everything (for releases).

We've built our own cross-compilation docker image with the go std library prebuilt for each platform, as well as installing the necessary cross-compilation toolchain for each arch.

We also rely on go's build tags in several places to choose platform-appropriate modules, which is something that is not currently supported by Bazel's go rules AIUI. (We could possibly use select() rules, though that's a little ugly.)

We've got Bazel working, but it pretty much only does the right thing if you build on a linux/amd64 host right now.

Kubernetes has no C I think. Kubelet depends on some linux specific headers.

Any update on this? How to build a .exe with a linux_amd64 environment?

Sorry for joining the conversation so late. I joined the Go team a short time ago, and I'll be working on improving Bazel support. Cross compilation is a priority for us, and I'll be working on this in the near future.

To echo what has been said elsewhere, @pcj's PR #98 is a short term solution if you need something working now.

In the long term, I agree with @yugui's comment: it's probably not a good idea to have separate rules for cross compilation. This can add redundancy and complication if a library is depended on in multiple configurations, or if the same library or binary needs to be built in multiple configurations. Ideally, when you declare a target, you should be saying what to build, not how to build it. The configuration should come from --cpu or --host_cpu or something similar.

It sounds like Bazel makes this difficult at the moment. I'll look into improving this.

AFAICT it's made difficult as the go rules use the linker from CROSSTOOL, not the go native ones, so there needs to be a valid C++ toolchain to get this going in go as presently written.

Is there any workaround at this moment?

@junghoahnsc Work on this is blocked for now. We're waiting for the new Bazel support for platforms. I expected we'll have some level of support in late Q2 or early Q3.

I expect we'll have some level of support in late Q2 or early Q3.

How's this going? I guess it is quite early in Q3, so it might be too soon to expect much yet.

@ianthehat has been working on this. Nearly all of the work is done already in the rules, but we need a few final pieces to land in Bazel to support platforms and toolchains. We're hoping that at least makes it into an RC some time in July, but it could slip further. Once that lands, it should only take a few more days to enable cross compilation on our side.

Note that we will probably need to drop support for older versions of Bazel when we start using these new features.

I have just been told I can start testing an early (not in the Bazel repository yet) preview of the features we need.
This probably means we are at least a few weeks out from finishing the feature, and then we will need to wait for a release of Bazel with it all in before we can enable it, but progress is being made.
I will update this issue when I have more news.

Cross compilation is now working using the new toolchains.
You need the very latest HEAD of bazel as of today, and also #625, but then you can cross compile linux from mac.
Further cross compilation pairs will be forthcoming, at the moment the limitations are in the cc support not the go rules, we could enable pure go cross compilation for any pair of host/target that the go compiler supports easily.

Very exciting! Thank you all for your work on this and particularly for the ongoing updates on this issue, @ianthehat and @jayconrod.

This sounds great!

If it's not too much to ask, would you consider prioritizing cross-compiles targeting ARM architectures? I know many folks use Go to cross-compile to smaller ARM devices (e.g. a Raspberry Pi). I suspect a GOARCH/GOOS of arm/linux will be one of the most common targets. I love using Bazel for my projects where I can, and I'd love to be able to use it for my stuff that targets my ARM devices.

@BranLwyd We'll try to get arm builds working as soon as we can, but Darwin/Linux amd64 builds are a really common use case, and we're focusing on that first. We use standard Go SDKs, which have built-in support for arm, so I imagine we'll be capable of building pure Go projects for arm soon. Building cgo and C++ code for arm will be more of a challenge though.

With the current system, we can set up the toolchains for all valid go cross compilation targets trivially, but there are two issues.

The first is that we have to pre-build the standard library for cross compilation toolchains as we download the go sdk, without knowing if they are ever going to be used.
In theory we can fix this by creating a proper dependancy that builds the toolchain on demand, but that's not easy to do, as the way it's built right now cannot succeed inside the sandbox environment of a normal build. Still deciding what the right way to work around this one is, until then I am only going to add cross compilation targets on demand so I don't cause huge first time build costs for people that are never going to cross compile.

The second issue is that we have to depend on the cc fragment for cgo support, and this is an unconditional dependancy, it happens even if you don't have any cgo in your project. This means that we require a valid cc cross compile toolchain to exist as well, and the only one I have found right now is mac->linux. This is much harder to fix, partly because cc does not support toolchains yet, partly because the cc rules are still implemented directly inside bazel (rather than in skylark) and partly because we are the go team, so we are dependant on other teams to make it work.

I could try to make it so that even when there is no valid cc toolchain you can still cross compile pure go code, and then enable linux->arm standard library building, not totally sure it is possible but if that would help I can at least research it.

If you could file a new issue for me with exactly what you need (and ideally a project I can test the rules on :) ), as this one has too much stuff in it and I would like to close it as soon as basic cross compilation lands, that would be great.

Dear @jayconrod @ianthehat ,
It was said that in the 0.5.4 release, cross compilation would be supported and we could create the Linux executable in Mac machine, kindly could you please comment on it, like is that it is implemented because I wanted to do cross-compilation and I'm not pretty much clear how could I implement it and use the Cross-toolchains in my project.
Thanks in advance!

Cross compilation is launched as of this Monday.
You need Bazel 0.5.4 and the master branch of rules_go.

bazel build --cpu k8

when compiling on Mac will build a linux binary. More to come.

Trying to do mac->linux cross build + statically link the binary. Getting this:

bazel  build --cpu=k8 //cmd/smith
WARNING: /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/io_bazel_rules_go/go/def.bzl:167:3: DEPRECATED: go_repositories has been deprecated. go_rules_dependencies installs dependencies the way nested workspaces should, and go_register_toolchains adds the toolchains.
INFO: Found 1 target...
ERROR: /Users/ash2k/gopath/src/github.com/atlassian/smith/cmd/smith/BUILD.bazel:20:1: GoLink cmd/smith/smith failed (Exit 1).
/private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/go1_9_darwin_amd64/pkg/tool/darwin_amd64/link: running external/local_config_cc/cc_wrapper.sh failed: exit status 1
ld: warning: option -s is obsolete and being ignored
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

2017/08/31 21:39:46 error running linker: exit status 1
Target //cmd/smith:smith failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.232s, Critical Path: 2.09s

Bazel 0.5.4
rules_go: master at 2e319588571f20fdaaf83058b690abd32f596e89
Same error with both Go 1.8.3 and 1.9.
WORKSPACE:

git_repository(
    name = "io_bazel_rules_go",
    remote = "https://github.com/bazelbuild/rules_go.git",
    commit = "2e319588571f20fdaaf83058b690abd32f596e89",
)
git_repository(
    name = "io_bazel_rules_docker",
    remote = "https://github.com/bazelbuild/rules_docker.git",
    tag = "v0.1.0",
)

load("@io_bazel_rules_go//go:def.bzl", "go_repositories")
load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories")

go_repositories(
    go_version = "1.9",
)

docker_repositories()

Rule:

go_binary(
    name = "smith",
    gc_linkopts = [
        "-linkmode",
        "external",
        "-extldflags",
        "-static",
    ],
    importpath = "github.com/atlassian/smith/cmd/smith",
    library = ":go_default_library",
    visibility = ["//visibility:public"],
)

I would have been amazed if this worked, I didn't think that clang can do static linking while cross compiling on mac... I don't think it has the required information?

As a small aside, you know you can do

bazel build --output_groups=static //cmd:smith

to build a static binary?

@ianthehat Thank you for your quick turnaround. I've executed the command in two ways.
1. As suggested "bazel build --cpu k8 //..." and getting the below error.
VJ:audit vijay.dugini$ clear
VJ:audit vijay.dugini$ bazel build --cpu k8 //...
INFO: Reading 'startup' options from /Users/vijay.dugini/audit/.bazelrc: --batch
INFO: (08-31 10:04:58.793) Loading package: src/build/pypi
INFO: (08-31 10:05:18.990) Loading package: @bazel_tools//tools/cpp
INFO: (08-31 10:05:19.054) Loading package: @local_config_xcode//
ERROR: (08-31 10:05:19.318) No toolchain found for cpu 'k8'. Valid cpus are: [
darwin,
armeabi-v7a,
x64_windows_msvc,
x64_windows_msys,
s390x,
ios_x86_64,
].
INFO: (08-31 10:05:19.319) Elapsed time: 21.185s

2. bazel build --cpu armeabi-v7a //...
With this command the build executes fine but the executable which is not supposed to run in MAC as it is build for Linux is still running in MAC, I assume which is unexpected behaviour.

Do I need to add any rule in my project WORKSPACE. Please advice.
TIA

@ianthehat that for the tip. I've just tried it and it does not work for me (note that this is not even a mac->linux build):

bazel build --output_groups=static //cmd/smith
WARNING: /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/io_bazel_rules_go/go/def.bzl:167:3: DEPRECATED: go_repositories has been deprecated. go_rules_dependencies installs dependencies the way nested workspaces should, and go_register_toolchains adds the toolchains.
INFO: Found 1 target...
ERROR: /Users/ash2k/gopath/src/github.com/atlassian/smith/cmd/smith/BUILD.bazel:20:1: GoLink cmd/smith/smith.static failed (Exit 1).
/private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/go1_9_darwin_amd64/pkg/tool/darwin_amd64/link: running external/local_config_cc/cc_wrapper.sh failed: exit status 1
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

2017/09/01 16:41:17 error running linker: exit status 1
Target //cmd/smith:smith failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.350s, Critical Path: 2.21s

Also, regarding the original issue - I actually want to build a docker image using rules_docker:

docker_build(
    name = "docker",
    entrypoint = ["/smith"],
    files = [":smith"],
)

So if I do this:

bazel build --cpu=k8 --output_groups=static //cmd/smith:docker
WARNING: /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/io_bazel_rules_go/go/def.bzl:167:3: DEPRECATED: go_repositories has been deprecated. go_rules_dependencies installs dependencies the way nested workspaces should, and go_register_toolchains adds the toolchains.
INFO: Found 1 target...
Target //cmd/smith:docker up-to-date (nothing to build)
INFO: Elapsed time: 1.567s, Critical Path: 0.00s

Nothing seems to be produced at all. Same with

bazel build --output_groups=static //cmd/smith:docker

I guess it works as intended (bazel newbie here)?

--output_groups selects a set of output groups produced by the target rule on the command line. docker_build probably doesn't have a static output group, so that won't produce anything. It will still grab the files from the default (non-static) output group of :smith.

For a rule to get a non-default output files of another rule, you have to go through a filegroup.

docker_build(
    name = "docker",
    entrypoint = ["/smith"],
    files = [":smith_static"],
)

filegroup(
    name = "smith_static",
    srcs = [":smith"],
    output_group = "static",
)

This probably won't work if you were unable to build the static binary directly from the command line. As Ian mentioned, there are problems linking static binaries on Mac, and these problems may extend to cross-compiling.

Couple additional questions:

  • Do you have any cgo code or other native code that gets linked in? I think we always have cgo turned on, but it may not be necessary to use the external linker when cgo is turned off.
  • Are you able to cross-compile this binary using GOOS=linux GOARCH=amd64 go build ...?

Dear @ianthehat , my issue has been resolved.
Before trying the cross compilation stuff, initially got to clear the cache as it holds the previous files which would throw error in building the file.

sudo rm -R /private/var/tmp/_bazel_$USER/
bazel build --cpu=k8 //...

I think that is a bug in the bazel auto-cc toolchain logic
From the previous list you posted it looks like it did not add k8 to the list of supported cpu's in an older version of Bazel, and then because it is a repository rule with no dependencies that changed, it never looked again, but support was added in a newer version of Bazel. It might be worth reporting it to the Bazel team as a separate issue.

having to do a bazel clean after upgrading bazel doesn't sound unreasonable (though maybe that could be the default behavior when it detects an upgrade/older artefacts)

@jayconrod thanks for explaining!

Do you have any cgo code or other native code that gets linked in? I think we always have cgo turned on, but it may not be necessary to use the external linker when cgo is turned off.

Not in my code, but I use client-go, apimachinery, etc from Kubernetes. Maybe some of the transitive dependencies have cgo libs. How can I find out?

Are you able to cross-compile this binary using GOOS=linux GOARCH=amd64 go build ...?

Yes. Both with and without CGO_ENABLED=0 compiles well. Interestingly having that flag does not change binary size. I thought it should be bigger with it?

FYI another thing that does not seem to work is cross compilation with -race:

bazel build --cpu=k8 //cmd/smith:smith-race 
WARNING: /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/io_bazel_rules_go/go/def.bzl:167:3: DEPRECATED: go_repositories has been deprecated. go_rules_dependencies installs dependencies the way nested workspaces should, and go_register_toolchains adds the toolchains.
INFO: Found 1 target...
ERROR: /Users/ash2k/gopath/src/github.com/atlassian/smith/cmd/smith/BUILD.bazel:27:1: GoLink cmd/smith/smith-race failed (Exit 1).
warning: unable to find runtime.a
warning: unable to find runtime/race.a
/private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/go1_9_darwin_amd64/pkg/tool/darwin_amd64/link: cannot open file /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/go1_9_darwin_amd64/pkg/linux_amd64_race/context.a: open /private/var/tmp/_bazel_ash2k/ea451d98a8bdaad3872b7aa995bea02c/external/go1_9_darwin_amd64/pkg/linux_amd64_race/context.a: no such file or directory
2017/09/04 09:43:43 error running linker: exit status 1
Target //cmd/smith:smith-race failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.506s, Critical Path: 0.35s
go_binary(
    name = "smith-race",
    gc_linkopts = ["-race"],
    library = ":go_default_library",
    visibility = ["//visibility:public"],
)

I'm not surprised, we would need to also build the standard library in race mode, which we don't, and I don't really want to impose the burden of automatically building in race mode on all users during repository download, I don't think the benefit of cross compiling race mode binaries is worth the cost.
I am trying to make it so the standard library is treated the same was as any other library, built on demand using the rules_go rather than part of the sdk, if I manage that then this will start working for free (along with building any cross compiled standard library on demand)

For what it's worth, there is an easier way to build race binaries built in to the rules as well, don't use the gc_linkopts.
You can build race binaries with

bazel build --output_groups=race

and run tests in race detection mode with

bazel test —features race 

This has the advantage of being actions that won't happen unless you deliberately trigger them, so you won't have rules that don't work in cross compile mode in your system, plus you don't have to maintain two separate binary rules.

Neat! It's great to have this feature. I'm trying to compile a Linux binary from an OS X machine. The build succeeds (with many warnings) but the output seems to be a malformed OS X binary rather than the Linux binary. Builds on both platforms (without cross compilation) work without issue.

Has anyone seen this before? I've included the build output and some followup commands below. Thanks!


Build Output

$ bazel build --experimental_ui --cpu k8 //...
............
INFO: Analysed 14 targets.
INFO: Found 14 targets...
INFO: From Linking external/com_github_google_protobuf/libprotobuf_lite.a [for host]:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/host/bin/external/com_github_google_protobuf/libprotobuf_lite.a(arenastring.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/host/bin/external/com_github_google_protobuf/libprotobuf_lite.a(atomicops_internals_x86_msvc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/host/bin/external/com_github_google_protobuf/libprotobuf_lite.a(io_win32.o) has no symbols
INFO: From Linking external/com_github_google_protobuf/libprotobuf.a [for host]:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/host/bin/external/com_github_google_protobuf/libprotobuf.a(gzip_stream.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/host/bin/external/com_github_google_protobuf/libprotobuf.a(error_listener.o) has no symbols
INFO: From Compiling external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
           ^
1 warning generated.
INFO: From Linking external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(_cgo_export.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_env_unset.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_file_unix.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_bluetooth_linux.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_constants.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_dev_linux.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_dirent.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_endian_little.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_env_unix.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_flock.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_race0.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_sockcmsg_linux.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_sockcmsg_unix.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_str.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall_linux.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall_linux_amd64.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall_linux_amd64_gc.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall_unix.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_syscall_unix_gc.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_zerrors_linux_amd64.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_zsyscall_linux_amd64.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_zsysnum_linux_amd64.cgo2.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo(external_org_golang_x_sys_unix_ztypes_linux_amd64.cgo2.o) has no symbols
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: warning for library: bazel-out/darwin_x86_64-fastbuild/bin/external/org_golang_x_sys/unix/libgo_default_library.cgo_c_lib.lo the table of contents is empty (no object file members in the library define global symbols)
INFO: From Linking external/org_golang_x_sys/unix/go_default_library._cgo_.o:
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
INFO: From GoLink cmd/cryptdo/cryptdo:
ld: warning: option -s is obsolete and being ignored
ld: warning: ignoring file /tmp/go-link-058981668/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /tmp/go-link-058981668/go.o
INFO: From GoLink cmd/cryptdo-filter/cryptdo-filter:
ld: warning: option -s is obsolete and being ignored
ld: warning: ignoring file /tmp/go-link-220379614/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /tmp/go-link-220379614/go.o
INFO: From GoLink cmd/cryptdo-rekey/cryptdo-rekey:
ld: warning: option -s is obsolete and being ignored
ld: warning: ignoring file /tmp/go-link-776447501/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /tmp/go-link-776447501/go.o
INFO: From GoLink cmd/cryptdo-bootstrap/cryptdo-bootstrap:
ld: warning: option -s is obsolete and being ignored
ld: warning: ignoring file /tmp/go-link-198729286/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /tmp/go-link-198729286/go.o
INFO: From GoLink go_default_xtest:
ld: warning: option -s is obsolete and being ignored
ld: warning: ignoring file /tmp/go-link-738500262/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /tmp/go-link-738500262/go.o
INFO: Elapsed time: 137.086s, Critical Path: 11.72s
INFO: Build completed successfully, 272 total actions



file reports that this is a OS X binary:

$ file bazel-bin/cmd/cryptdo/cryptdo
bazel-bin/cmd/cryptdo/cryptdo: Mach-O 64-bit executable x86_64

Attempting to run the binary results in an error:

$ ./bazel-bin/cmd/cryptdo/cryptdo
dyld: lazy symbol binding failed: Symbol not found: _main
  Referenced from: /Users/x/workspace/cryptdo/./bazel-bin/cmd/cryptdo/cryptdo
  Expected in: flat namespace

dyld: Symbol not found: _main
  Referenced from: /Users/x/workspace/cryptdo/./bazel-bin/cmd/cryptdo/cryptdo
  Expected in: flat namespace

Abort trap: 6

When might we see support for cross-compiling from linux_amd64 to linux_arm? Is there work still to be done is Bazel before the Go rules can support it?

@xoebus
That's weird. I suspect this is a cgo issue, can you try a pure go cross compile first so we can make sure. For instance, the following is a record of me compiling and then cross compiling one of our examples:

$bazel build //examples/bin
INFO: Found 1 target...
Target //examples/bin:bin up-to-date:
  bazel-bin/examples/bin/bin
INFO: Elapsed time: 3.212s, Critical Path: 0.56s

$file bazel-bin/examples/bin/bin
bazel-bin/examples/bin/bin: Mach-O 64-bit executable x86_64

$bazel build --cpu=k8 //examples/bin
INFO: Found 1 target...
Target //examples/bin:bin up-to-date:
  bazel-bin/examples/bin/bin
INFO: Elapsed time: 0.730s, Critical Path: 0.39s

$file bazel-bin/examples/bin/bin
bazel-bin/examples/bin/bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

@dangerousHobo
The only work needed in the bazel rules should be a single line to add the standard library cross compilation, but the same should be true for all valid go cross compiles, the limitation now is the existence of a valid cc crosstool. I am looking it to ways to fix both these issues.

Cross compiling the standard library on demand is quite a lot of work, and going to take a while, but does not have anything I am aware of that would block me from doing it. I filed #814 so you can track that work.

I am also looking at writing a crosstool stub that would enable at least all pure go cross compiles, even if cgo did not work.

@ianthehat You're right. Something's odd with cross compilation (with standard go tool) with one of my libraries even though at a quick glance they should work. I'll dig in further.

I cleaned up some of the dependencies and updated to the latest (HEAD) rules version. The new error I get is:

$ bazel build --cpu k8 //...
INFO: Found 14 targets...
ERROR: /Users/user/workspace/cryptdo/cryptdopb/BUILD.bazel:9:1: GoCompile cryptdopb/~normal~go_default_library~/code.xoeb.us/cryptdo/cryptdopb.a failed (Exit 1).
bazel-out/darwin_x86_64-fastbuild/bin/cryptdopb/code.xoeb.us/cryptdo/cryptdopb/cryptdo.pb.go:16:8: import /private/var/tmp/_bazel_user/314e6db7a04fa07463806a43d1e0e1ca/bazel-sandbox/4863999288350891313/execroot/cryptdo/bazel-out/host/bin/external/com_github_golang_protobuf/proto/~normal~go_default_library~/github.com/golang/protobuf/proto.a: object is [darwin amd64 go1.9 X:framepointer] expected [linux amd64 go1.9 X:framepointer]
2017/09/18 11:16:56 error running compiler: exit status 1
INFO: Elapsed time: 0.520s, Critical Path: 0.10s

I get the protocol buffers dependency transitively from the Go proto rules. Am I making a mistake that causes this library not to be cross compiled? I can push this WIP somewhere if that would help.

With the new cross-compile support, how can I tell go_binary to build for target linux_386, linux_amd64, or windows?

@xoebus That error looks like it is still picking up an archive from the wrong platform somehow. Specifically it's picking up a host build of the protobuf library when trying to compile a linux library.

I can think of no way you could make this happen... If you can stick it somewhere I can get it I can take a look.

@ianthehat Thank you - I appreciate it. The changes are now on the master branch of xoebus/cryptdo.

Was this page helpful?
0 / 5 - 0 ratings