Rules_go: specific SHA has interoperability issue with rules_docker

Created on 29 Jun 2019  ·  8Comments  ·  Source: bazelbuild/rules_go

What version of rules_go are you using?

http_archive(
    name = "io_bazel_rules_go",
    urls = ["https://github.com/bazelbuild/rules_go/archive/8ea79bbd5e6ea09dc611c245d1dc09ef7ab7118a.tar.gz"],
    sha256 = "039c159e8f16a365a5f0ce31c8e6ca8acba5708962e904c1f164ba79fb8efe4d",
    strip_prefix = "rules_go-8ea79bbd5e6ea09dc611c245d1dc09ef7ab7118a",
)

What version of gazelle are you using?

http_archive(
    name = "bazel_gazelle",
    urls = ["https://github.com/bazelbuild/bazel-gazelle/releases/download/0.17.0/bazel-gazelle-0.17.0.tar.gz"],
    sha256 = "3c681998538231a2d24d0c07ed5a7658cb72bfb5fd4bf9911157c0e9ac6a2687",
)

What version of Bazel are you using?

tested 0.27.0 -> 0.23.1

Does this issue reproduce with the latest releases of all the above?

Using the latest

What operating system and processor architecture are you using?

darwin, amd64

Any other potentially useful information about your toolchain?

Trying to get cross compilation working.

What did you do?

See https://github.com/chrislovecnm/external-dns/tree/work-on-bazel

What did you expect to see?

Cross compiled go binary running in a container.

What did you see instead?

ERROR: While resolving toolchains for target //:external_dns_image: no matching toolchains found for types @io_bazel_rules_docker//toolchains/docker:toolchain_type
ERROR: Analysis of target '//:external_dns_image' failed; build aborted: no matching toolchains found for types @io_bazel_rules_docker//toolchains/docker:toolchain_type

I have tried rules_docker 0.7.x -> 0.8.1 branches and I am still experiencing the same issues. I have tried lower versions of bazel and have had the exact same issues. So the cross compile for go, works like a champ! But add rules_docker, and we get the no matching toolchain issue.

See https://github.com/bazelbuild/rules_docker/issues/943

tl;dr

Using latest bazel and a SHA from go_rules, cross build build --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 works like a champ. Add rules_docker to the mix, and I am getting a toolchain error.

bug cgo cross-compilation go
Source
picture chrislovecnm
👍1

Most helpful comment

@nlopezgi

I can say that lots of users are developing on osx with containers, IMO supporting this use case is important.

@jayconrod the interesting thing I am seeing is that the toolchain reporting that we have the darwin_constraint is is_darwin even though we are compiling for linux 🤔.

picture chrislovecnm on 2 Jul 2019
👍2

All 8 comments

I took a quick look at this and was able to reproduce the problem. I'm not sure why this is happening. The --debug_toolchain_resolution output is hard to read in this case.



$ bazel build --toolchain_resolution_debug --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 :external_dns_image

Starting local Bazel server and connecting to it...

INFO: Invocation ID: 2cedf5d5-b31c-486b-a842-2db90adc51d4

Loading:

Loading: 0 packages loaded

Analyzing: target //:external_dns_image (1 packages loaded, 0 targets configured)

INFO: ToolchainResolution: Selected execution platform @bazel_tools//platforms:host_platform,

INFO: ToolchainResolution: Looking for toolchain of type @io_bazel_rules_docker//toolchains/docker:toolchain_type...

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @bazel_tools//platforms:os has value @bazel_tools//platforms:windows, which does not match value @bazel_tools//platforms:linux from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @bazel_tools//platforms:os has value @bazel_tools//platforms:osx, which does not match value @bazel_tools//platforms:linux from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @bazel_tools//platforms:os has value @bazel_tools//platforms:windows, which does not match value @bazel_tools//platforms:linux from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   Considering toolchain @docker_config//:toolchain...

INFO: ToolchainResolution:     Toolchain constraint @bazel_tools//platforms:os has value @bazel_tools//platforms:osx, which does not match value @bazel_tools//platforms:linux from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:darwin_constraint has value @io_bazel_rules_go//go/toolchain:is_darwin, which does not match value @io_bazel_rules_go//go/toolchain:not_darwin from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:     Toolchain constraint @io_bazel_rules_go//go/toolchain:cgo_constraint has value @io_bazel_rules_go//go/toolchain:cgo_on, which does not match value @io_bazel_rules_go//go/toolchain:cgo_off from the target platform @io_bazel_rules_go//go/toolchain:linux_amd64

INFO: ToolchainResolution:   Rejected toolchain @docker_config//:toolchain, because of target platform mismatch

INFO: ToolchainResolution:   No toolchains found

ERROR: While resolving toolchains for target //:external_dns_image: no matching toolchains found for types @io_bazel_rules_docker//toolchains/docker:toolchain_type

ERROR: Analysis of target '//:external_dns_image' failed; build aborted: no matching toolchains found for types @io_bazel_rules_docker//toolchains/docker:toolchain_type

INFO: Elapsed time: 2.747s

INFO: 0 processes.

FAILED: Build did NOT complete successfully (13 packages loaded, 147 targets configured)

FAILED: Build did NOT complete successfully (13 packages loaded, 147 targets configured)

rules_go has defined some new platforms, toolchains, and constraints at master. Previously, rules_go always required a C/C++ toolchain, even if it wasn't used. It was fine to use a toolchain that was incompatible with the target platform because we didn't use it for cross-compilation. Since bazel 0.26.0, the C/C++ toolchain is selected with the platform and toolchain system, which means that either a compatible C/C++ toolchain must be provided, or no C/C++ should be required.

To get around this, I added a new constraint, @io_bazel_rules_go//go/toolchain:cgo with values cgo_on and cgo_off. The default value is cgo_on so that if the target platform is the default platform, cgo will be enabled. Platforms like linux_amd64 set cgo_off. Go toolchains compatible with cgo_off do not request a C/C++ toolchain.

Bazel seems to be indicating that the Docker toolchain isn't compatible with the constraint values from the linux_amd64 platform. I'm not sure why though because the Docker toolchain doesn't mention these constraints, so there shouldn't be a compatibility issue.

cc @katre I may have some questions for you tomorrow, but let me know if anything is obviously wrong.

jayconrod picture jayconrod on 2 Jul 2019

/cc @nlopezgi

Lions tigers an toolchains ... err bears 😁

@jayconrod afiak rules_docker does not need the C/C++ toolchain either, as it is python internally. So I am confused as well.

—-

Update: there is go and python, so they may need c for cgo, but I am guessing not. Nick will know better

chrislovecnm picture chrislovecnm on 2 Jul 2019

I was also able to successfully reproduce the error. But I'm not quite sure we can do anything on rules_docker to avoid it.
It seems like cross-compiling go on mac for linux is really the use case we are looking for here to be well supported in order for go_image to be usable from a mac. I should note, this has never been explicitly called out as a use case that is supported by rules_docker / go_image, it just happened to be the case that for a little bit of time it was possible to trick bazel / rules_go into doing this properly. I think that if doing this (and setting up tests to keep it working) is not too much effort from rules_go, then lets do it, otherwise, I'm also prepared to accept that this is just not a supported use case for go_image (i.e., building / running a go_image from mac is not supported at all).

nlopezgi picture nlopezgi on 2 Jul 2019

@nlopezgi

I can say that lots of users are developing on osx with containers, IMO supporting this use case is important.

@jayconrod the interesting thing I am seeing is that the toolchain reporting that we have the darwin_constraint is is_darwin even though we are compiling for linux 🤔.

chrislovecnm picture chrislovecnm on 2 Jul 2019
👍2

We set cgo_on and is_darwin (on macOS only) as default values for cgo_constraint and darwin_constraint. This is so that when you're targeting the default platform (@bazel_tools//platforms:target_platform) which doesn't specify any value for these, you get the correct values.

It appears that setting default_constraint_value in constraint_setting not only adds that value to a platform that doesn't specify another value... it also adds that value to target_constraints of a toolchain that doesn't specify another value. So the Docker toolchains require the default values and do not work with any platform that asks for something else.

I've filed bazelbuild/bazel#8778. I don't think this is the intended behavior.

I'll try and think of an alternative way for rules_go to enable cgo.

jayconrod picture jayconrod on 2 Jul 2019

@jayconrod thanks!!

chrislovecnm picture chrislovecnm on 2 Jul 2019

@nlopezgi

I can say that lots of users are developing on osx with containers, IMO supporting this use case is important.

I understand that this is important to users, but it still seems to me Bazel might not be fully ready to support it properly. So I'd rather set up the expectations (and the maintenance responsibilities) in a way that is clearer for users and easier for us to deal with. I'll be happy to try this again once its resolved, but we should only announce it as supported once we have set up testing in all repos involved (upstream bazel, rules_go, rules_docker) so that if a solution is working it does not regress.

nlopezgi picture nlopezgi on 2 Jul 2019

@nlopezgi I agree that we need to test this :) . Completely agree

chrislovecnm picture chrislovecnm on 2 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dlorenc picture dlorenc  ·  7Comments
rickypai picture rickypai  ·  5Comments
apesternikov picture apesternikov  ·  3Comments
senior7515 picture senior7515  ·  4Comments
cdelguercio picture cdelguercio  ·  3Comments