Bazel: Support macOS Big Sur and Apple Silicon hardware platform

[philwo]

This is a tracking bug for adding support for macOS Big Sur and the Apple Silicon platform to Bazel. I'm currently trying to acquire a DTK to run some initial tests and will follow up here when I find out more.

We aim to have full support in time for their final release.

[keith]

This may be out of the scope of this discussion, but I think it would be nice to break the assumptions in bazel that if you're compiling for an Apple CPU you have to be running on darwin_x86_64 (previously, and now darwin*). This would unlock cross-compilation to Apple platforms from Linux and I think could be done without degrading the UX for macOS users. This would also likely make it easier to add more CPUs like this one.

[keith]

I also have a DTK coming and would be happy to contribute some here as this is a pre-req for us long term as well.

[kayasoze]

My DTK has arrived and I've hit this roadblock. Happy to help.

[dmaclach]

Just to be clear to what roadblock are you referring?

[philwo]

Just FYI, we're still waiting for the DTK, so no news from my side yet.

[dmaclach]

FYI

https://github.com/bazelbuild/bazel/commit/b235517a1f3b91561a513ea4381365dba30dd592#diff-1f4964536f397e8679625954c7139bd6

and

https://github.com/bazelbuild/bazel/commit/e4e06c0293bda20ad8c2b8db131ce821316b8d12#diff-1f4964536f397e8679625954c7139bd6

add support for darwin_arm64 and darwin_arm64e respectively.

blaze build --config=darwin_arm64 should get you a basic Apple Silicon binary.

[dmaclach]

I think that #12035 should unblock folks. Apologies on my earlier assertions as I was using a different crosstool and forgot that my changes there were not available externally. We will leave this bug open for the time being until we have bazel working natively on Apple Silicon, but for the time being you should be able to build on x86_64 for arm64(e) and build (running an emulated bazel) on arm64 for arm64(e).

[keith]

Related https://github.com/bazelbuild/rules_apple/issues/980

Any general update on this from google folks? Is anyone actively looking at this? I could probably spend some time on this soon, I have a DTK

[BalestraPatrick]

We are also blocked on being able to target arm64 and the iOS simulator. I also have a DTK but I'm unfamiliar with Bazel core, but happy to help in any way I can.

[keith]

AFAIUI currently we're blocked on the JDK not supporting Apple Silicon. Realistically after that I imagine the fact that homebrew doesn't support it either will make it difficult to build and test this.

[BalestraPatrick]

@keith Isn't the fact that Bazel doesn't support Apple Silicon a different problem? With my understanding, I assumed we should be able to still pass the right arguments to the compiler and build targeting arm64 and the iOS simulator even if Bazel isn't yet optimized with Apple Silicon.

[keith]

Sorry I thought you were hoping to do the build on the DTK as well. Yes we should be able to target the apple silicon simulator from x86_64 bazel runs. I don't think this will be an easy change based on a discussion at bazelcon (not sure if you were in the apple BoF). I think the problem is there isn't a way to differentiate simulator or device for the same architecture.

[BalestraPatrick]

Yeah, I was in the discussion so I'd love to know how big of a change this would be from some Google folks, since it could block Bazel adoption relatively soon for teams who want/need to support this new architecture.

[keith]

Looks like there's a zulu release for Apple Silicon https://www.azul.com/downloads/zulu-community/?os=macos&architecture=arm-64-bit&package=jdk

[keith]

@philwo given the blocker around simulator vs device builds, do googlers have an opinion on the solution? I'm interested in working on this but want to submit something that you all will be happy with as well.

[philwo]

@keith I _think_ we'd be more than happy to take your design proposal / contribution on this 😊, but let's check whether there are any conflicts or opinions here - @dmaclach @susinmotion any concerns or opinions on this from your side?

The relevant context is how https://github.com/bazelbuild/rules_apple/issues/980 should be dealt with, if I understand correctly.

[philwo]

@keith FYI, @meteorcloudy and me are looking into getting Bazel to build with the Apple Silicon native JDK. Thanks for pointing out that Azul has a Zulu release for this!

[keith]

Happy to hear you're working on the build! Once it's ready I'm definitely interested in trying it!

[susinmotion]

Yes, please, that would be amazing! I think this is related to some work that @gregestren 's team is doing; looping him in.

[gregestren]

@katre is there a relation to platforms or execution environments here?

[katre]

@gregestren There shouldn't be any need for changes to Bazel's platforms or toolchains machinery to support this. I'm happy to consult on using those facilities if needed.

[philwo]

Pasting some notes here, so I don't forget:

If we just run Bazel with the Apple Silicon Zulu, loading the JNI libraries will not work, because they're x86_64.

So we have to use an x86_64 Bazel (with x86_64 Zulu) and cross-compile Bazel first, which "should" work via:

bazel build --cpu=darwin_arm64 --host_cpu=darwin_arm64 //src:bazel

However, @meteorcloudy pointed out that this could lead Bazel to selecting the wrong sources for the mac build (as we use "darwin" only in the select statements). So we'll need something like this, but for Bazel: https://github.com/meteorcloudy/bazel/commit/3139666084e4bed7979822d8f13b99fea8f5aa20

Regarding arm64 vs. arm64e, the latter one is more modern, but it appears that Zulu is built for arm64 - which is good (for now), because that doesn't require us to codesign every binary. Apparently that's only required for the newer arm64e binaries. If we do want to go to arm64e eventually, we'll have to codesign -v -s - --force my_lib.dylib/my_binary everything we build somehow.

[keith]

I wonder if we should replace the darwin conditional with one that means any of the darwin architectures as opposed to splitting out all cases? for example https://github.com/envoyproxy/envoy/pull/6182/files

[meteorcloudy]

@keith Yes, that's what I did in https://github.com/meteorcloudy/bazel/commit/3139666084e4bed7979822d8f13b99fea8f5aa20 with skylib's config_setting_group

[katre]

See @comius's recent work on using constraint values instead of flags: 6d637f4165445a0d6644f6e6f8a61b3f1de208d8

[thii]

I've successfully built an arm Bazel with https://github.com/bazelbuild/bazel/pull/12594.

[thii]

I spoke too soon 😢 Although it's an arm binary, host tools were built for x86_64 so Bazel still invokes everything under Rosetta.

[keith]

Based on the comments above do we need to update some selects?

[thii]

The Java code that generates HOST_CONSTRAINTS returns x86_64 for the current CPU, so I'm _guessing_ it's because of the embedded JDK.

[keith]

Can probably update that? https://github.com/bazelbuild/bazel/issues/11628#issuecomment-728222211

[thii]

It works!

There's no minimal arm64 JDK published yet so the output binary size is 200 MB. I wonder why minimizing JDK couldn't be done as part of the build.

[philwo]

I wonder why minimizing JDK couldn't be done as part of the build.

I would love it, if we could do that. It would simplify our processes when updating a JDK so much. Unfortunately the last time we looked into it, jlink wasn't deterministic, so the build of Bazel would no longer be deterministic.

Looking at the upstream bug (I think it was https://bugs.openjdk.java.net/browse/JDK-8214230), it looks like that may have been fixed in newer JDKs! So we could try again if it's deterministic now.

FYI @meisterT

[meisterT]

Oh, wow. Having that fixed would indeed be nice. There's one more problem though that I see with running this as part of the build: it takes a whole lot of time to minimize. I don't want to add this into the critical path of our end-to-end tests, but also would like to test the real thing.

[thii]

If anyone wants to try it out, 013a24216b96c12484e1a38ddaf5c24a9cdff12f can build you a Bazel 4.0.0rc2 binary that targets Apple Silicon.

USE_BAZEL_VERSION=4.0.0rc2 bazelisk build -c opt --cpu=darwin_arm64 --stamp --embed_label 4.0.0rc2 //src:bazel

Update: The full patch needed is https://github.com/bazelbuild/bazel/compare/37a429ad12b4c9e6a62dbae4881a1ff03b81ab40...thii:4.0.0-darwin_arm64?expand=1.

[indragiek]

For anyone who needs to use bazel to build an arm64 slice for running the iOS Simulator on Apple silicon, I posted a workaround here: https://github.com/bazelbuild/rules_apple/issues/980#issuecomment-738645357

[thii]

See @comius's recent work on using constraint values instead of flags: 6d637f4

This seems to break cross-compiling (as it was mentioned in the commit message). What's the proper way to cross-compile Bazel now? (I've tried to create a platform target and build with that but it failed to detect C++ toolchain.)

[thii]

@philwo Can you help build and publish the minimal JDKs for arm64? I think they're the only missing pieces now. The remaining issues are specific to each ruleset.

[erinacio]

Manually patched https://github.com/bazelbuild/bazel/pull/12035 to version 2.2.0, successfully compiled an dawin-arm64 version of bazel 2.2.0 with zulu8 (arm64 build) and it seems to work. Tried a simple C++ project (stage3 of bazel C++ tutorial project) and it produced arm64 program without any issue. Maybe it's possible to officially build some darwin-arm64 binaries for old versions of bazel?

[ayankumarh]

I am trying the command.

USE_BAZEL_VERSION=last_green bazelisk build -c opt --cpu=darwin_arm64 //src:bazel

It fails with the following error :-

`WARNING: Running Bazel server needs to be killed, because the startup options are different.
Starting local Bazel server and connecting to it...
DEBUG: /private/var/tmp/_bazel_ayanhalder/41cd2b7fe220c72c9678bf9436ba6009/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:59:14: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '3.1.0'

DEBUG: /private/var/tmp/_bazel_ayanhalder/41cd2b7fe220c72c9678bf9436ba6009/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True 

DEBUG: /private/var/tmp/_bazel_ayanhalder/41cd2b7fe220c72c9678bf9436ba6009/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:59:14: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '3.1.0'

DEBUG: /private/var/tmp/_bazel_ayanhalder/41cd2b7fe220c72c9678bf9436ba6009/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True 

ERROR: /private/var/tmp/_bazel_ayanhalder/41cd2b7fe220c72c9678bf9436ba6009/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin_arm64'

ERROR: Analysis of target '//src:bazel' failed; build aborted: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Elapsed time: 10.325s
INFO: 0 processes.

FAILED: Build did NOT complete successfully (118 packages loaded, 736 targets configured)`

Please let me know what I am missing.

[Johannes-Lange]

@erinacio could you upload your 2.2.0 bazel darwin arm binaries?

[erinacio]

@erinacio could you upload your 2.2.0 bazel darwin arm binaries?

Maybe nope. It's actually not so hard to build one for yourself, and GitHub doesn't support uploading binary files to issue page.

[jwbee]

Please let me know what I am missing.

I see the same behavior as @ayankumarh which suggests that those of you who have this working have something else in your repo that isn't available to the rest of us. I can't get it working with any combination of the patches mentioned here.

[jesec]

It is surprisingly easy. I successfully built the latest commit (bbeb1da) on my M1 Mac mini.

1. Download bazelisk from https://github.com/bazelbuild/bazelisk/releases. darwin-amd64 is fine. Install it to /usr/local/bin/bazelisk.
2. Download and install JDK 11 (.dmg) from https://www.azul.com/downloads/zulu-community/?os=macos&architecture=arm-64-bit&package=jdk.
3. Clone Bazel repo: git clone [email protected]:bazelbuild/bazel.git --depth=1 --single-branch
4. Inside the repo, USE_BAZEL_VERSION=last_green bazelisk build -c opt --apple_platform_type=macos --cpu=darwin_arm64 //src:bazel

That's it.

Target //src:bazel up-to-date:
  bazel-bin/src/bazel
INFO: Elapsed time: 780.524s, Critical Path: 183.13s
INFO: 2985 processes: 179 internal, 2093 darwin-sandbox, 1 local, 712 worker.
INFO: Build completed successfully, 2985 total actions

https://drive.google.com/file/d/13b5_sd4d8LrZS18aVq8K2EOuxtPNmuWL/view?usp=sharing

[jesec]

https://github.com/bazelbuild/bazel/issues/11628#issuecomment-737048107

I spoke too soon 😢 Although it's an arm binary, host tools were built for x86_64 so Bazel still invokes everything under Rosetta.

Um.. This happens as well.