Zig: Tier 1 Support for ARM 64-bit (aarch64) Linux

[andrewrk]

Once all these items are complete, we can say that Zig has Tier 1 support for 64-bit ARM Linux.

• [x] make sure we can build-exe with -target aarch64v8-linux-gnu
• [x] make sure we can build-exe with -target aarch64v8-linux-musleabi
  
  
  
  • lld: error: undefined symbol: __floatsitf - I believe this is the last one! #1290
  
  
• [x] make sure stack traces / debug info works & add test coverage
• [x] add the target to the test matrix
• [ ] [get the full test suite passing](https://github.com/ziglang/zig/issues?q=is%3Aopen+label%3Aarch-arm64+label%3Abug)
• [x] figure out a 64-bit arm linux continuous integration testing service and hook it up to zig's CI testing.

[sepruitt]

Hello, I'm interested in this because I'm attempting an Android port of zig. I see that there are at least two free AArch64 linux CI services for OSS projects, Shippable and Drone. One of those can probably be set up once the test suite passes.

[SamTebbs33]

Hello, I'm interested in this because I'm attempting an Android port of zig. I see that there are at least two free AArch64 linux CI services for OSS projects, Shippable and Drone. One of those can probably be set up once the test suite passes.

Thanks for finding those @sepruitt , I'll take a look at them once we have the tests passing, which I'm currently setting up a docker image for.

[shawnl]

So things have gone backwards with the addition of libuserland.a. Better support of the C ABI is now necessary in order to get things working again. http://infocenter.arm.com/help/topic/com.arm.doc.ihi0055b/IHI0055B_aapcs64.pdf

[andrewrk]

@shawnl Thanks for the notice. Looks like doing #661 would serve us well right now.

We do have C ABI tests in test/stage1/c_abi/ but right now the tests are disabled for non x86_64 architectures:

https://github.com/ziglang/zig/blob/d83b15febf00518e201160f8b089c0d846d18e4c/test/build_examples.zig#L26-L28

So I think the path forward here is

• [ ] #661
• [ ] get the C ABI tests passing for aarch64
• [ ] add C ABI tests for aapcs calling convention

[markfirmware]

FYI https://github.com/WorksOnArm/cluster
I might be able to help.

[SamTebbs33]

FYI https://github.com/WorksOnArm/cluster
I might be able to help.

Thanks @markfirmware, looks like we'll have to submit an access request similar to https://github.com/WorksOnArm/cluster/issues/165

[shawnl]

The biggest problem I ran into when trying to do this is the high memory requirements of the stage1 compiler, which means a much more expensive VPS.

[markfirmware]

I can see that @shawnl has a ball rolling over at #2978.

[shawnl]

@markfirmware I am spending 6e/mo however, and it would be nice to be able to use donated time instead.

[daurnimator]

I have access to worksonarm and am setting up a CI runner.

[SamTebbs33]

What's the CI status here? I'd like to help get arm support progressing.

[andrewrk]

Most CI services do not have arm64 support.

I tried SourceHut's arm64 support but it's qemu based and took over 3 hours to even finish compiling. That's too slow. I spoke with Drew DeVault about this and the cost of arm64 server hardware was the main reason native arm64 is not available. He also ran a poll and found that outside of zig nobody really needed arm64 testing.

@daurnimator tried setting up a GitLab CI runner using the docker installation approach and ran into an open GitLab bug for arm64 that was a showstopper.

@shawnl has a shell-based GitLab runner instance running and it's on my to-do list to figure out how to integrate it into the CI.

[daurnimator]

@daurnimator tried setting up a GitLab CI runner using the docker installation approach and ran into an open GitLab bug for arm64 that was a showstopper.

I'm waiting for a response from @solidnerd via @vielmetti on a fix for this.

The bug was https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725#note_198579090 currently assigned to @tmaczukin

@shawnl has a shell-based GitLab runner instance running and it's on my to-do list to figure out how to integrate it into the CI.

The issue with that approach is that it becomes very hard to "reset" the machine between runs. IMO it would be suitable for testing master, but not PRs from arbitrary users.
If you do have a solution to this, I've got the hardware (donated by WorksOnArm)

[SamTebbs33]

Since we have the hardware, all we theoretically need is the build server that will interface with GitHub to build PRs. We discussed a few options on IRC and I suggested TeamCity, but Jenkins could be the best solution.

You can make it run for PRs and there's a docker image available that could run on the arm hardware. Running in docker could solve the "resetting" problem. I haven't experimented with this at all since I don't have my own arm hardware to run it on at the moment.

[andrewrk]

Good news -

With the merge of #3290, we are much closer to tier 1 support for Arm-64 bit linux. We now have userland qemu-based Continuous Integration test coverage for:

• behavior, std lib, and compiler-rt tests for aarch64v8_5a-linux-none
• behavior, std lib, and compiler-rt tests for aarch64v8_5a-linux-musl, statically linking against a zig-built musl
• behavior, std lib, and compiler-rt tests for aarch64v8_5a-linux-gnu, dynamically linking against a zig-built glibc

However, some tests are disabled. The remaining issues to solve are:

• #3288
• #3282
• #2934

Once these are solved, and we add test coverage for stack traces on aarch64 linux, 64-bit ARM linux can be promoted to a Tier 1 Supported target.

[vielmetti]

A couple of recently announced CI resources that might be of use here for arm64 builds:

Drone Cloud has arm64 native support available, announced at https://blog.drone.io/drone-announces-official-support-for-arm/

Travis CI now has arm64 support available as well, with an announcement at https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

In both cases you can connect this Github project codebase to these CI systems and test PRs as they happen - that might be a very suitable path to explore.

[andrewrk]

Drone Cloud is working great. We now have binaries for ARM 64 bit available on the download page: https://ziglang.org/download/, and CI is hooked up.

So now it's down to getting all the tests passing. See the issues labeled "arch-arm64". When these are all solved, we should be able to run the full test suite in the CI script instead of only the behavior tests:

https://github.com/ziglang/zig/blob/e219f0c786e8982c81d089b2945891632028b6ef/ci/drone/linux_script#L23-L27

[vielmetti]

It's been a few months since I checked in here; I see in #4457 there is at least one issue. How is it going for aarch64 Tier 1 support and what can I do to help?

[andrewrk]

We are quite close. It will be time to re-evaluate this when the llvm10 branch is merged, which should solve most of the remaining issues. LLVM is about to cut rc4 and I expect that one to actually get released. Likely within 1-2 weeks.

[andrewrk]

There are still some open issues for aarch64, which you can find by searching for the corresponding label, but even so, I think it's clear that this target now qualifies for Tier 1 Support.

I've updated ziglang.org accordingly.

Big thanks to Drone CI for the CI service and WorksOnARM for the beefy test box that we've been using to troubleshoot aarch64 issues. For example, I believe @LemonBoy took advantage of that machine in a heroic effort to solve https://github.com/ziglang/zig/pull/4832.

[andrewrk]

Sorry, I'm going to have to walk this back a little bit.

Tier 2 says:

Some tests may be disabled for these targets as we work toward Tier 1 support.

This is correct, and not planned to change the definitions of tiers. So it remains to get the disabled tests passing before we can claim Tier 1 support.