Runtime: Running dotnet core on Alpine for ARM32

[JensNordenbro]

Due to its small footprint (OS image size), Alpine is suited for embedded scenarios on ARM32.

Things leading to this:

• Today dotnet core is Community Supported on ARM32.
• dotnet core 2.1 will support arm32 officially (Supported Operating Systems and Chip Architectures-section)

• Today dotnet core 2.1 is running on Alpine for x86.
• It is possible to run docker on an embedded system (for example ResinOS) and in order to do this efficiently a small container image is needed, hence the need for Alpine support.
• It is expected to have symmetry in OS support between different CPU architectures where possible

(If/when this is implemented the docker images for running .net core application should be available. )

[RussKeldorph]

@Petermarcu @richlander @janvorli

[janvorli]

See my comment https://github.com/dotnet/coreclr/issues/9370#issuecomment-364912259
We want to support ARM32 Alpine, but in order to do that, we need a way to cross build for Alpine arm on x64. The cross build mechanisms that we have in our build now works only for debian based distros and uses qemu-debootstrap. That would not work for Alpine.
In the comment above, I've identified a potential way to do that.

[ghost]

Alpine arm32 architecture is called armhf in their docs, which supports armv6 and armv7 target architectures.

Unfortunately, they only provide armv6-alpine-linux-musleabi, which does not compile CoreCLR, as armv6 is not supported: https://github.com/dotnet/coreclr/issues/10605. I tried compiling coreclr by adding

elseif(CLR_CMAKE_PLATFORM_ALPINE_LINUX) 
  add_compile_options(-target armv6-alpine-linux-musleabihf)

in compileoptions.cmake, but compilation failed due to unsupported assembly instructions (dmb ish etc.)

I then searched and asked Alpine community over chat, apparently they considered/discussed it at some point but have "made a decision" to not provide the said armv7 official release; it is either armv6 for 32-bit or armv8 for 64-bit. This way they will not have to support two variants of armhf.

[ghost]

One approach could be to build our own compiler toolchain and armv7 ABI using https://github.com/richfelker/musl-cross-make, then ship it with the ARM32 runtime package. But I think then all the external native dependencies would need to be compiled with same toolchain? Sounds complicated.

[janvorli]

Oh, that's unfortunate. But as for your previous comment, why would we need to ship the toolchain? Wouldn't compiling the runtime bits with it be sufficient?

[ghost]

I was thinking that we might need to ship the generated ABI libs (not toolchain). It makes sense that technically it should be sufficient to build and compile in a way that we shouldn't need to package any extra libs like that with our bins (it should be just work on alpine installation on armv7 device).

Out of the box, they support GCC. It would be ideal for us to generate armv7-alpine-linux-musleabihf, then we can use clang to compile with this abi target. Found related conversation: https://github.com/richfelker/musl-cross-make/issues/34.

By the way, i used troyfontaine/armhf-alpinelinux (https://github.com/troyfontaine/armhf-alpinelinux) qemu alpine 3.7 docker to try compiling coreclr. Will test out musl-cross-make. Even with extra qemu-ish layers, it is super slim (6mbs)!

apk update
apk add cmake git bash build-base icu-dev lttng-ust-dev clang llvm-dev linux-headers
apk add lldb-dev --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing
git clone https://github.com/dotnet/coreclr
cd coreclr

# apply musl ABI patch in compileoptions

./build.sh -cmakeargs -DCMAKE_CXX_COMPILER=$(which clang++)

(for some reason, cmake picks up /bin/c++ instead of /bin/clang++ in this docker)

[janvorli]

(for some reason, cmake picks up /bin/c++ instead of /bin/clang++ in this docker)

I have seen cases when clang package has installed clang, but clang++-3.x or vice versa. In other words, the version was just on one of those and it was causing the same or similar issue. Adding a symbolic link to make it symmetric helped.

I was thinking that we might need to ship the generated ABI libs

I believe the ABI is the same and the only difference is in the supported instructions inside the binaries. So we should be able to build dotnet core using the self compiled toolchain, but the result should work on the Alpine as is.

[ghost]

I made a simple test:

#include <stdio.h>
int main()
{
   asm volatile ("dmb ish" : : : "memory");
   return 0;
}

compiled with clang test.c -o test.exe -march=armv7-a on that docker and it worked (without -march=armv7-a, it gives the error). It seems like something is wrong with CMake configuration (something is overwriting CFlags).

My current patch is:

diff --git a/compileoptions.cmake b/compileoptions.cmake
index 9c352e8b..f87bfea4 100644
--- a/compileoptions.cmake
+++ b/compileoptions.cmake
@@ -66,6 +66,9 @@ if(CLR_CMAKE_PLATFORM_UNIX_ARM)
      add_definitions(-DARM_SOFTFP)
      add_compile_options(-mfloat-abi=softfp)
      add_compile_options(-target armv7-linux-gnueabi)
+   elseif(CLR_CMAKE_PLATFORM_ALPINE_LINUX)
+#     add_compile_options(-target armv7-alpine-linux-musleabihf)
+     set(CMAKE_C_FLAGS "-march=armv7-a ${CMAKE_C_FLAGS}")
    else()
      add_compile_options(-target armv7-linux-gnueabihf)
    endif(ARM_SOFTFP)

[ghost]

Pushed one more commit to https://github.com/dotnet/coreclr/pull/17730, with that CoreCLR builds on ARM32 Alpine using steps described https://github.com/dotnet/coreclr/issues/17468#issuecomment-382831060.

[ghost]

I have experimented with chroot and finally made it work:

Using docker to get Ubuntu with nested virtualization enabled

docker run -it --privileged ubuntu:bionic

entered Ubuntu docker:

# update packages
apt update

# install some utils
apt install -y apt-utils wget < /dev/null

# download alpine chroot script (and verify its checksum)
wget https://raw.githubusercontent.com/alpinelinux/alpine-chroot-install/v0.8.0/alpine-chroot-install \
  && echo 'a3d7e2e3e63dfb8abcabb35829a6c8c18bdab082  alpine-chroot-install' | sha1sum -c

# make the script executable
chmod 777 alpine-chroot-install

# call script (see other options in that script, like to mount a directory, use `-i` option)
./alpine-chroot-install -a armhf -b v3.7 -m http://dl-cdn.alpinelinux.org/alpine

# enter chroot
/alpine/enter-chroot

entered alpine armv7 chroot:

# update packages
apk update

# install build prerequisites and utilities
apk add cmake git bash build-base icu-dev lttng-ust-dev clang llvm-dev linux-headers

# install lldb, which is in testing repository
apk add lldb-dev --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing

# clone coreclr
git clone https://github.com/dotnet/coreclr
cd coreclr

# build
./build.sh -cmakeargs -DCMAKE_CXX_COMPILER=$(which clang++)

@janvorli, could you please help integrating it in CoreCLR's cross build scripts? The additional toolchain.cmake etc. are confusing, as there we are again setting ABI which is not strictly necessary (imo for other platforms as well)

[janvorli]

@kasper3 that is awesome! Thank you so much for looking into it and making it work!
I'll be happy to do the integration.

[ghost]

One upstream issue is fixed: https://github.com/alpinelinux/alpine-chroot-install/issues/8, I have updated steps above to use v0.8.0 version of chroot script.

[sdmaclea]

@janvorli Would you be willing to do this for arm64 also. I think alpine is interesting in arm64 docker scenarios because it is lightweight.

[janvorli]

@sdmaclea sure, I think there would be only minor differences needed for arm64.

I'll get to it next week, I'm flying home from Redmond this afternoon.

[janvorli]

Update: I have the build working for both arm and arm64, but I need to test it on a real device / container yet to make sure the compiled binaries are 100% correct.

@kasper3 the steps you have provided were really a great guideline. However, they were actually compiling under the qemu and so the compilation was really slow. So I have implemented the cross build the same way as we build for debian / ubuntu. That means that the build system's clang is used for building and the rootfs just provides the libraries / headers to compile / link against.
The only issue was that the LD sets the .interp section to point to the glibc loader instead of the musl one, since it knows nothing about musl. For the time being, I've added a linker option that asks it to set it explicitly to the musl one, but I am not sure if that's a correct solution.
I am sorry for the delay, but I was OOF this and the last week Monday and Tuesday.

If you are interested, you can check the current state here:
https://github.com/janvorli/coreclr/commit/0a03ff8f51b87fc8c93b274aa370fef1d5f91192

[ghost]

Thanks @janvorli. It looks good to me, but if possible, I think it would be nice to request @rofl0r for the pull request review to get some holistic feedback. 😎

A bit off-topic, should we try to merge Android rootfs script into the same one to easily keep the implementation for all arm platforms in sync?

[ghost]

CoreCLR part is done in dotnet/coreclr#18234. Should this be closed from CoreCLR and open in CoreFX / coreSetup for remaining work?

[janvorli]

CoreFX PR for the same thing is pending: https://github.com/dotnet/corefx/pull/30182. I'll make the same change in core-setup soon.

[janvorli]

And here core-setup: https://github.com/dotnet/core-setup/pull/4203

[JensNordenbro]

Will this work be for 2.1 lts or 3.0?

[sdmaclea]

Definitely not for 2.1. 2.2 or 3.0

[janvorli]

@sdmaclea why not for 3.0? We have not discussed it yet, but 3.0 seems like a very reasonable target.

[ghost]

Thanks @janvorli!
I think the remaining work is to get CLI story for ARM32 set: https://github.com/dotnet/cli/issues/8998? It seems like @echesakovMSFT hasn't committed to it yet, do you know what are the predicaments in crossgening arm32 on x64?

[sdmaclea]

Definitely not for 2.1. 2.2 or 3.0

@sdmaclea why not for 3.0?

The period after the 2.1 was too subtle. Should have said "Definitely not for 2.1. Either for 2.2 or 3.0"

[janvorli]

Ah, makes sense.

[echesakovMSFT]

@kasper3 Answering your question about crossgening arm32 on x64 - I was distracted during 2.1 shipping. Now it has been shipped and I am back actively working on this problem.

[janvorli]

@kasper3 actually, there is more to do. First we need to enable official builds in coreclr, corefx and core-setup repos so that nuget packages are built and published.

[JensNordenbro]

Any progress on this?

[mshgh]

@janvorli just curious has the target been discussed? Any chance for this to make it into 2.2? ;) Or at least the 3.0 mentioned above as "likely very reasonable target"?

[richlander]

It is unclear to me what is left to discuss here ...

Both the .NET Core 2.1 runtime and SDK are now working on Linux+ARM32, including Alpine. Last time I checked, there were still bugs on Linux+ARM64 for .NET Core 3.0, but Linux+ARM64 is targeted for that release.

[mshgh]

@richlander I have checked the 2.1 download page and it seems to me the version requested in this issue is still not available. But, honestly, I'd be more then happy if you prove me wrong.

When looking into the Linux section it is not immediately obvious to me which package should be downloaded when. Let me explain. There are two sections Linux glibc and Linux musl this is great, but this doesn't tell you which architecture these packages are for. Then if you check the package names this is 2.1.0-linux-x64 and 2.1.0-musl-x64 respectively. This reveals architecture - Intel 64bits. Note the first package would be better named 2.1.0-glibc-x64. And there is no 32bit version, which is not big deal.

Now there is another section called Linux ARM with 32/64bit packages named 2.1.0-linux-arm and 2.1.0-linux-arm64. If I follow the naming convention chosen for Intel this would be glibc based => 2.1.0-glibc-armXX, but there is not musl versions for these packages. And Alpine is based on musl. So package I am looking for is 2.1.0-musl-arm, respectively to use the name for architecture as Alpine packages do it the name would be 2.1.0-musl-armhf.

In the end the names are not critical, if the binaries deliver what is necessary. And the milion dollar question is. Does such package exist?

P.S. Yes, on 32bit ARM Alpine can be installed glibc support in parallel to musl and then the 2.1.0-linux-arm package should work, but this is not what I am looking for. and I am sure neither author of this issue.

[JensNordenbro]

I have the same experience and also there seems to be no docker Images available.

[janvorli]

@richlander we still don't have official builds for Alpine ARM / ARM64. So we have neither SDK nor CLI yet. In the past, I have just enabled the crossbuild support for Alpine ARM / ARM64 in our repos, but I haven't added the official builds.

[mshgh]

@janvorli no pressure ;) but the follow up question is obvious. Are there any plans to add the official builds for Alpine ARM?

[hjc4869]

Vote +1 for official build. I test my code using linux-arm64 NuGet packages everyday even though it's not yet stable and officially supported. It's much more convenient to have official linux-musl-arm(64) packages published.

[JensNordenbro]

Vote +1; @janvorli ; Having the runtime is good for many scenarios; sdk + cli + offical docker images can follow later.

[hjc4869]

It seems that Alpine ARM64 will be officially supported in .NET Core 3.0. Unfortunately Alpine ARM32 is not on the list.

[mshgh]

Good catch @hjc4869, unfortunately I cannot see any link to "ARM64 Alpine" in the .NET 3.0 downloads page. Only "x64 Apline" one is there. Or did you have better luck locating the binaries?

[hjc4869]

Binary and docker images are not released yet, I guess they'll come later.

[janvorli]

@hjc4869 I am planning to enable Alpine for both arm32 and arm64. @mshgh we have to enable building Alpine ARM/ARM64 nuget packages yet.

[mshgh]

.NET Core 3.0 is supposed to be released pretty soon. Are there any plans to make this part of the 3.0?

[galvesribeiro]

Updates on that folks?

[JensNordenbro]

@janvorli, there seems to be x64 musl available at https://github.com/dotnet/core-setup/blob/master/README.md, is arm32/64 so special considering this + arm32/64 for glibc is already impl. or is the delay just about prioritization?

[janvorli]

@JensNordenbro it is a matter of enabling build and testing of related packages in our infrastructure.
We are currently in a process of enabling ARM64 musl packages build and publishing.
@richlander can you please comment on what are our timings / plans w.r.t. enabling ARM32 Alpine package builds?

[JensNordenbro]

Is this issue out or in for .Net core 3.x?

[AddRemover]

I am looking at any build for musl arm32 (not arm64 available on 3.0).
Since appears it's not yet officially supported.
Anyone can provide me a link for dotnet-runtime-linux-musl-arm32?

[ghost]

Are there instructions/toolchain available to build 2.2 for armhf (arm32) now? Is it straight forward now?
Of course, I'd prefer binaries, but have no hope to find them in foreseeable future...