Prysm: Add support for cross compilation in Bazel

[prestonvanloon]

Currently, we're unable to build for other OS / architecture whenever the binary target depends on a cgo library. This includes any dependency on go-ethereum in the beacon-chain and potentially a future BLS library.

To reproduce the issue, try building the beacon chain with a different toolchain specified.

Building in on linux targeting windows:

bazel build --platforms=@io_bazel_rules_go//go/toolchain:windows_amd64 //beacon-chain

Building on linux amd64 targeting arm64.

bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64 //beacon-chain

Important caveats:

• We're using a fork of go-ethereum which has the generated BUILD files to support bazel.
• Removing cgo dependencies are not an option as we expect to have more cgo dependencies in the future. Pure go builds would be nice and we'd have support for windows / ARM from linux today if that were the case!
• We must support linux -> [darwin_amd64, arm64, windows_amd64] targets. When you're writing cc toolchains, adding Mac OS as the host is optional, but appreciated (expect a bounty tip if Mac OS as host is supported!).
• This is a pretty involved task. If you are up to taking it on, you'll be mostly on your own to figure it out. Good luck :)

These blocked issues may have some additional context from previous attempts to resolve this.

Blocking #654
Blocking #2546

[katzdan]

Id' like to get into this one. Wanted to delve into Bazel for a while now.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__This issue now has a funding of 600.0 DAI (600.0 USD @ $1.0/DAI) attached to it.__

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
  
  
  
  • Want to chip in? Add your own contribution here.
  
  
  • Questions? Checkout Gitcoin Help or the Gitcoin Slack
  
  
  • $135,704.07 more funded OSS Work available on the Gitcoin Issue Explorer
  
  

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__Workers have applied to start work__.

These users each claimed they can complete the work by 3 months, 3 weeks from now.
Please review their action plans below:

1) suburbandad has applied to start work _(Funders only: approve worker | reject worker)_.

Using https://github.com/emacski/tensorflow-serving-arm as an example, build cross compile configs for linux-arm64 and windows-x86_64, leveraging https://github.com/dockcross/dockcross as the build environment. The net result should be that any host execution with x86_64 architecture should be able to cross compile for windows_x86-64 and linux_arm64 targets.
2) thecydonian has applied to start work _(Funders only: approve worker | reject worker)_.

I will first familiarize myself with the existing code, then I will research solutions to the problem. This seems potentially challenging, but I am willing to take it on, and I will not give up easily. If I am able to solve this, I will be sure to do so for MacOS as well.

I will make sure to keep you consistantly updated on my progress, and should be able to make signifigant progress over the weekend.

In the unlikely case that I do fail at fixing this issue. I will produce a write up with everything I found out while trying to do this bounty. Thank you for your consideration

Learn more on the Gitcoin Issue Details page.

[prestonvanloon]

@lzxm160, it has to be bazel. We might be adding more complicated build logic that wouldn't be supported natively with gomod.

[lzxm160]

@prestonvanloon sure,I'll try if you approve

[prestonvanloon]

@lzxm160, I took a quick look at your profile on gitcoin and you have 3 abandoned tasks. I wouldn't want to award this task to you so that no one else has the opportunity to work on it then have it abandoned later.

Do you have a plan on how to do this? If you already know the solution then I'd accept it and award the bounty, but this is something that won't be easy for someone who isn't familiar with bazel toolchains.

[lzxm160]

@prestonvanloon understand,I'll try first

[prestonvanloon]

To anyone interested in working on this, please build from branch spec-v0.6 unless https://github.com/prysmaticlabs/prysm/pull/2312 is merged to master.

We've added some progress towards this goal with #2970. You can see the TODO here: https://github.com/prysmaticlabs/prysm/blob/25d8b485e6d45deaa001f877265f266466623f4b/tools/binary_targets.bzl#L1-L9

Suggested workflow here would be to uncomment those lines then build the following from linux:

• //beacon-chain:beacon-chain-darwin-amd64
• //beacon-chain:beacon-chain-windows-amd64

If both of those targets build on linux, then this issue can be resolved.

[3ach]

I tried building the darwin_amd64 and windows_amd64 rules on my machine and it worked fine. is this still an issue?

[prestonvanloon]

This is resolved by toolchains in rules_go.

[prestonvanloon]

Reopening this as we still have cross compile issues.

[prestonvanloon]

I have this script to build and upload multiple platform binaries here: https://gist.github.com/prestonvanloon/32fdec43aba8bd96c656739ada69a23e#file-build_and_upload-sh

To consider this issue resolved, we should be able to run the above script on linux and successfully build all four targets:

Host machine: linux amd64

Targets:

• linux amd64
• linux arm64
• darwin amd64
• windows amd64

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__The funding of this issue was increased to 1000.0 SAI (1000.0 USD @ $1.0/SAI) .__

• If you want to claim the bounty you can do so here
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $90,148.83 more funded OSS Work available on the Gitcoin Issue Explorer

[SuburbanDad]

I am working on a cross-compiling docker container and toolchains that will support cross compiling arm64, darwin and windows from any x86-64 machine that has docker installed.

So far I have docker images and toolchain branches for :

arm64/linux/clang10 -> successfully builds prysm beacon-chain and validator
x86_64/darwin/clang9 -> partial success; builds bls target but failing on beacon and validator
x86_64/win/clang10 -> no progress to report yet
Good progress, but still a ways out before a PR

[prestonvanloon]

Awesome progress @SuburbanDad.

If possible, can you check in the ARM build? We can start providing arm64 docker images sooner

[SuburbanDad]

If possible, can you check in the ARM build? We can start providing arm64 docker images sooner

I will get it into a committable state tonight and publish the MVP docker cross compiler image :+1:

[SuburbanDad]

PR submitted for all four working targets. 🎉

[prestonvanloon]

Update: @SuburbanDad's PR meets all of the requirements of the bounty. In PR review, I asked if we could make use of bazel's remote build execution and/or docker sandbox. Over a few days of offline discussion, @SuburbanDad and I were able to connect the plumbing from PR #4945 into the bazel RBE and bazel docker sandbox functionality for cross compile.

So far, we are able to cross compile and confirm the following:

| target | linux_amd64 | linux_arm64 | osx_amd64 | windows_amd64 |
|----------|-------------------|------------------|-----------------|-----------------------|
| //beacon-chain | :heavy_check_mark: docker-sandbox and RBE, libkafka supported locally only | :heavy_check_mark: docker-sandbox and RBE, no libkafka support | :heavy_check_mark: docker-sandbox, no libkafka support | :heavy_check_mark: docker-sandbox, no libkafka support |
| //validator| :heavy_check_mark: docker-sandbox and RBE | :heavy_check_mark: docker-sandbox and RBE | :heavy_check_mark: docker-sandbox | :x: Doesn't work. #5008 |

@SuburbanDad and I will work together to merge our changes into PR #4945 by the end of the week.This will enable us to leverage RBE in our continuous integration and more importantly provide signed executable binaries for each tagged release of Prysm's beacon-node and validator targets. :tada:

[prestonvanloon]

@ceresstation This is good to go! Please release the bounty to @SuburbanDad

[gitcoinbot]

⚡️ A tip worth 1000.00000 SAI (1000.0 USD @ $1.0/SAI) has been granted to @SuburbanDad for this issue from @ceresstation. ⚡️

Nice work @SuburbanDad! Your tip has automatically been deposited in the ETH address we have on file.

• $73411.40 in Funded OSS Work Available at: https://gitcoin.co/explorer
• Incentivize contributions to your repo: Send a Tip or Fund a PR
• No Email? Get help on the Gitcoin Slack

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__This Bounty has been completed.__

Additional Tips for this Bounty:

• ceresstation tipped 1000.0000 SAI worth 1000.0 USD to SuburbanDad.

---

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $57,197.71 more funded OSS Work available on the Gitcoin Issue Explorer