Runtime: Enabling Linux ARM32 for .NET Core

[hqueue]

Host: Ubuntu 14.04 x64
Target: Ubuntu 14.04 or(and) 16.04 ARM, Ubuntu 14.04 ARM Softfp and Tizen

To enable core-setup for Linux ARM32, we are going to follow steps suggested in PR dotnet/core-setup#712.

ARM hardfp

Common

• [x] stable default clang version for cross compilation (#1411)
• [x] Use same logic to prepare rootfs across dotnet (#1432)

CoreCLR

• [x] Update CoreCLR to generate nuget packages of CoreCLR (PR dotnet/coreclr#8421)
• [x] Remaining CoreCLR ARM32 issues tracked by issue dotnet/coreclr#8549. Still on going

CoreFX

• [x] Add RID for Linux/ARM (dotnet/corefx#14161)
• [x] Add Xenial (Ubuntu 16.04) support for build-rootfs.sh (issue https://github.com/dotnet/corefx/issues/14538)
• [x] Make clang3.6 default toolset for cross arm32 build (issue https://github.com/dotnet/corefx/issues/15511)
• [x] Enable CI support for Linux Arm32 (https://github.com/dotnet/coreclr/issues/9273, https://github.com/dotnet/corefx/pull/15900)
• [x] Update CoreFX to generate nuget packages of CoreFX (issue dotnet/corefx#14298)
• [x] Enable CoreFX Pipeline build support for Linux for Arm32 and publish packages.

core-setup
For core-setup, we will follow steps below.

• [x] Add support for Linux Arm32 RID (PR dotnet/core-setup#712)
• [x] Preparing ROOTFS for cross-compilation Infrastructure (issue dotnet/core-setup#729, PR dotnet/core-setup#747)
• [x] Enable cross compiling corehost for Linux/ARM (issue dotnet/runtime#2547, PR dotnet/core-setup#749, dotnet/core-setup#766, dotnet/core-setup#812 )
• [x] Update build script to enable cross building (PR dotnet/core-setup#712)
• [x] Update build script to enable downloading and packaging for Linux/ARM (PR dotnet/core-setup#712)
• [x] Build nuget packages for the host and Microsoft.NETCore.App metapackage (PR dotnet/core-setup#712)
• [x] Make clang3.6 default toolset for cross arm32 build (issue https://github.com/dotnet/core-setup/issues/1322)
• [x] Enable CI support for Linux Arm32 (issue https://github.com/dotnet/core-setup/issues/790)(PR https://github.com/dotnet/core-setup/pull/1512)
• [x] Enable Pipeline build support for Linux Arm32 (issue https://github.com/dotnet/core-setup/issues/789)
• [x] Update Core-Setup readme.md to reflect the download location of Linux Arm32 archive binaries.

For example, we want to build core-setup for Linux/ARM using following command.

$ ./build.sh  --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs-coreclr/arm/

ASP.NET

• [x] Build LibUV for Arm32 (Issue https://github.com/aspnet/libuv-build/issues/19).

ARM softfp

We will use armel for arm-softfp through out the dotnet.

Common

• [x] stable default clang version for cross compilation (#1411)
• [x] Use same logic to prepare rootfs across dotnet (#1432)

Debian.8

CoreFX - Part1

• [x] Add RID for armel (dotnet/corefx#14792)

CoreCLR

• [x] Use armel for arm-softfp through out the dotnet. (https://github.com/dotnet/coreclr/issues/8770, https://github.com/dotnet/coreclr/pull/8771)
• [x] Generate packages for armel RID (dotnet/coreclr#8827)
• [x] Prepare ROOTFS for cross-compilation Infrastructure (already available)

CoreFX - Part2

• [x] Use armel for arm-softfp through out the dotnet. (https://github.com/dotnet/corefx/pull/14803)
• [x] Prepare ROOTFS for cross-compilation Infrastructure (already available)

core-setup
For core-setup, we will follow steps below.

• [x] Use armel for arm-softfp through out the dotnet (#1025)
• [x] Generate nupkgs and tarball for debian.8-armel (#1025)

./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=debian.8-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel

Tizen 4.0.0

CoreFX - Part1

• [x] Add RID for Tizen

CoreCLR

• [x] Prepare ROOTFS for cross-compilation Infrastructure (https://github.com/dotnet/coreclr/pull/8781)
• [x] Enable PipeLine build support for Linux armel (Tizen) (dotnet/coreclr#9689)

CoreFX - Part2

• [x] Prepare ROOTFS for cross-compilation Infrastructure (https://github.com/dotnet/corefx/pull/14844)
• [x] Prepare host packages for Tizen.4.0.0-armel (https://github.com/dotnet/corefx/issues/16175)
• [x] Remove workaround to prevent armel package restore (https://github.com/dotnet/corefx/issues/16174)
• [x] Enable CoreFX Pipeline build support for Linux for armel (Tizen) and publish packages (dotnet/corefx#16335)

core-setup
For core-setup, we will follow steps below.

• [x] Prepare ROOTFS for cross-compilation Infrastructure (#1025)
• [x] Generate nupkgs and tarball for tizen.4.0.0-armel (#1025)
• [x] Enable Pipeline build support for Linux armel (Tizen) (#1566)

dotnet-ci

• [x] Add Tizen OS to Machine map (https://github.com/dotnet/dotnet-ci/pull/603)

./build.sh -c Release --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=armel,TARGETRID=tizen.4.0.0-armel,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/armel-tizen

Results

ubuntu.14.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz
ubuntu.16.04-arm (Latest build is available at https://github.com/dotnet/core-setup#daily-builds)
dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (Last updated on Jan 19)
dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz
debian.8-armel
dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz
tizen.4.0.0-armel
dotnet-tizen.4.0.0-armel.1.2.0-beta-001273-00.tar.gz

[gkhanna79]

CC @schellap @ramarag @janvorli

[janvorli]

As I have mentioned in the discussion in dotnet/core-setup#724, I believe that we should be consistent in the way we do cross build over our repos. That means to use rootfs and to do the cross build using clang and not GCC. The same way we do it in CoreCLR and CoreFX.
I hope we can reuse the logic from the build.sh in CoreCLR here.

[gkhanna79]

@hqueue @hseok-oh Is Ubuntu 14.04 an intended target for Arm32?

[hqueue]

@gkhanna79 AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04.

[hseok-oh]

@hqueue @gkhanna79 RaspberryPI2 used ubuntu 14.04. (but no longer maintained)
https://wiki.ubuntu.com/ARM/RaspberryPi

[gkhanna79]

@hseok-oh If Ubuntu 14.04 is not maintained and supported, is there a specific reason to support it for Linux Arm32? If not, it will simplify bunch of things (e.g. libcoreclrtraceptprovider.so builds just fine for 16.04 but needs additional work for 14.04).

What do you think?

[gkhanna79]

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

[gkhanna79]

@kouvel This is the Linux Arm32 main work-list issue.

[gkhanna79]

CC @schellap

[hqueue]

AFAIK default rid is ubuntu.14.04-arm for coreclr ARM and default ARM rootfs is trusty which is ubuntu 14.04

I don't know who chose those defaults or why. Could it be cause Tizen OS is based upon Ubuntu 14.04? do we build libcoreclrtraceptprovider.so for Tizen?

@gkhanna79
I don't think it's related to Tizen OS.
The most possible explanation will be that ubuntu 14.04 is selected because of devices (e.g. ARM Emulator, Raspberry pi 2 and etc.) which were used to bring up CoreCLR ARM.
Please check out https://github.com/dotnet/coreclr/issues/3805 for ARM emulator which is used at the very early stage of CoreCLR ARM32 bring up.

[gkhanna79]

Thanks for the explanation @hqueue. For now, we can continue to build for both 14.04 and 16.04 and see how far we will need the 14.04 implementation.

With regards to the emulator, I noticed that in CoreCLR repo, we bootup the emulator and setup the rootfs within it and then perform the build. This is different from the build instructions we have at https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md where it comprises of two steps:

1. Build rootfs for an architecture
2. Perform a cross build of the repo

We strive hard to ensure that CI builds the same way as any developer would. Since the above steps are how any developer in the community would be expected to build for Arm32, what do you think of updating the CI script to do just that and not rely on emulator to be present for performing the build?

Secondly, the emulator should only be required to deploy binaries and execute tests. I notice that we only run 22 tests within the emulator, even though the repo has 11K+ tests. Do these 22 tests add any value from the CI perspective (e.g. here is a log of a recent PR that shows the 22 tests that ran - https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_emulator_cross_debug_ubuntu_prtest/1080/consoleFull) since they dont seem to cover key scenarios where breaks maybe introduced?

[hqueue]

@gkhanna79 For the first question, I will call @sjsinju @wateret, since I'm not aware of the latest of ARM CI.

@sjsinju @wateret Can you please answer the first question above regarding CI and ARM emulator ?

For the second one, I also think tests are not sufficient to cover key sccenarios. We will look into them.

[wateret]

@gkhanna79
CI cross-builds the same way as any developers would. You can check out arm32_ci_script.sh#L227. The emulator is only for running tests. Of course, we still need pre-built rootfs to cross-build though.

One more thing that may be confusing is that the CI has arm-softfp rootfs only which cannot be created by 'any developers'. There is my work on CoreFX for enabling arm-hardfp(ubuntu 14.04). After it's done I will apply it to CoreCLR as well.

[jyoungyun]

@gkhanna79 I couldn't find where libuv.so is built. Could you give me an advice?

[qmfrederik]

@jyoungyun not sure if it's the way libuv is built for .NET, but for raspberry pi the instructions so far have been to download it from the libuv project & compile it:

See the these comments from @leemgs related to dotnet/coreclr#6321

rpi2@arm# sudo apt-get install gyp
rpi2@arm# wget http://dist.libuv.org/dist/v1.0.0-rc1/libuv-v1.0.0-rc1.tar.gz 
(The latest version is v1.9.1: http://dist.libuv.org/dist/v1.9.1/libuv-v1.9.1.tar.gz)
rpi2@arm# tar -xvf libuv-v1.0.0-rc1.tar.gz
rpi2@arm# cd libuv-v1.0.0-rc1/
rpi2@arm# ./gyp_uv.py -f make -Duv_library=shared_library
rpi2@arm# make -C out
rpi2@arm# sudo cp out/Debug/lib.target/libuv.so /usr/lib/libuv.so.1.0.0-rc1
rpi2@arm# sudo ln -s libuv.so.1.0.0-rc1 /usr/lib/libuv.so.1

That worked for me to get Kestrel up & running on Raspberry Pi; but note that newer versions of libuv are now available at http://dist.libuv.org/dist/

[gkhanna79]

CI cross-builds the same way as any developers would.

@wateret My point is that we should decouple cross building in CI from emulator. The nuance is that in CI, we setup rootfs from emulator and then perform cross-build while a regular developer will not. My suggestion is to fix the script to do the following:

1) Without the emulator, perform the cross build (e.g. cross/build-rootfs.sh arm)
2) Load the emulator only for running tests. Thus, if we were to decide that emulator may no longer be needed for testing for some reason, then cross-build continues to function as it is expected.

[gkhanna79]

@hqueue did you find anything interesting about running the 22 tests in CI?

[jyoungyun]

@qmfrederik Thank you for your answer. What you are talking about is how to download libuv.so directly. But I want to know where the libuv.nupkg is generated. The NETCoreApp is dependent on libuv module so I wanna generate libuv.nupkg for arm.

[wateret]

@gkhanna79

we setup rootfs from emulator and then perform cross-build while a regular developer will not.

Just make things clear, the emulator image is nothing but pre-built rootfs. Of course the emulator also needs it for running tests.

For the first one you suggested, basically I agree with you. However it takes pretty long to build rootfs with build-rootfs.sh. As far as I remember it took 1-2 hours on my linux machine. I'm not sure if it is acceptable.
For the second one, the emulator is loaded only for running tests already. Build does not run on the emulator. It cross-builds.

[hqueue]

did you find anything interesting about running the 22 tests in CI?

@gkhanna79 Those 22 tests are TCs where we observed regression frequently when brining-up CoreCLR for Linux ARM32. Therefore the coverage of 22 tests does not cover the code which is not modified freqeuently. However we cannot run all TCs (11K+) with ARM emulator becasue it took very long time and we faced timeout in CI environment.

I also think we need more TCs for CI, but not sure about how to choose candidates. Can you suggest any options ?

cc: @jyoungyun

[qmfrederik]

@jyoungyun I think the libuv NuGet package is located here: https://github.com/aspnet/libuv-package?files=1

[jyoungyun]

@qmfrederik Thank you! I will check that repository.

Finally, I succeeded in obtaining a dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz file, which confirmed that it works well on Raspberry PI2. I built the core-setup repository using the local nupkg source and these are the results built from CoreCLR and CoreFX(including https://github.com/dotnet/corefx/pull/14655 patch). If CoreCLR and CoreFX nupkgs are uploaded in the NugetServer or anywhere, we can generate the .NET Core Runtime for Ubuntu ARM. I have tested only on Ubuntu 14.04 and will be testing on 16.04 later.

jyoung@DXL-Workstation:~/git/dotnet/core-setup-jy/artifacts/ubuntu.14.04-arm/packages$ ls
total 33M
-rwxrw---- 1 jyoung jyoung  28K Dec 21 18:18 dotnet-deb-tool.1.0.1-t-beta-001206.nupkg
-rw-rw---- 1 jyoung jyoung 226K Dec 21 18:19 dotnet-hostfxr-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
-rw-rw---- 1 jyoung jyoung  17M Dec 21 18:19 dotnet-sharedframework-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
-rw-rw---- 1 jyoung jyoung  17M Dec 21 18:19 dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz
drwxrwx--- 2 jyoung jyoung 4.0K Dec 21 17:04 intermediate

pi@raspberrypi:~/Downloads/24919/shared/Microsoft.NETCore.App/1.2.0-beta-001206-00 $ ./dotnet hello.exe 
Hello World

[qmfrederik]

Awesome! If there's any way for you to share the NuGet packages/tarballs, I'd be happy to try them out on a RPi 2 & 3, and see how far I get with our .NET Core-based product :)

[gkhanna79]

I succeeded in obtaining a dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz file

This is excellent news @jyoungyun! Thank you all for driving this through :)

The LibUV package does come from ASP.NET repository. @Eilon Can you please advise which repo to enlist to build this package from?

[gkhanna79]

If CoreCLR and CoreFX nupkgs are uploaded in the NugetServer or anywhere

This is the work I am tracking to get done as part of the E2E pipeline build support across the three .NET Core repos (CoreCLR, CoreFX, Core-Setup).

[gkhanna79]

@wateret

As far as I remember it took 1-2 hours on my linux machine

That is interesting. For me, creating a new rootfs takes about 7 or so mins on my USB 3.0 SSD for both Trusty and Xenial builds. Everything else is cost of a regular cross build from there on. Hence, my question around why we need to pickup rootfs from emulator and not just fetch it everytime. Building upon this, to enable CI build for Ubuntu 16.04 arm, are you planning to add another emulator that will contain rootfs?

@hqueue

Can you suggest any options ?

Do you have any automation around tests on real HW? My current thinking is to evaluate doing so in favor of emulator int he foreseeable future.

[janvorli]

I can also confirm that creating a new rootfs takes a couple of minutes on my Linux box. It is probably related to the speed of the internet connection, since the process fetches all the necessary packages.

[Eilon]

The libuv source code is 3rd party and is here: https://github.com/libuv/libuv

The ASP.NET team's build process for compiling libuv is in our own repo here: https://github.com/aspnet/libuv-build/

And it references the libuv source code via a Git submodule: https://github.com/aspnet/libuv-build/tree/dev/submodules

And finally, the ASP.NET team has a repo for taking the libuv binaries and packaging them into a NuGet package: https://github.com/aspnet/libuv-package

[Eilon]

cc @moozzyk who knows the most about the libuv build.

[moozzyk]

What @Eilon said.
Packages for different OS/architectures (currently Windows Win32/x64/arm, Linux (64-bit), macOS) are built from this repo https://github.com/aspnet/libuv-build/ (which pulls libuv code as a submodule) and pushed to the myget feed as OS specific Microsoft.AspNetCore.Internal.libuv-* packages. The libuv-package repo is responsible for pulling the build packages and creating the über libuv package that contains bits for all supported architectures.

[jyoungyun]

@qmfrederik I attached a .NET Core Runtime tarball for Ubuntu 14.04 ARM. When I am ready for Ubuntu 16.04, I will share the file here. (Updated on 12/26 : I attached a .NET Core Runtime tarball for Ubuntu 16.04 ARM.)
dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz (Ubuntu 14.04 ARM)
dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz (Ubuntu 16.04 ARM)

@gkhanna79

This is the work I am tracking to get done as part of the E2E pipeline build support across the three .NET Core repos (CoreCLR, CoreFX, Core-Setup).

Thank you for your support!

@Eilon @moozzyk @qmfrederik Thank you for comments. In fact, we do not need libuv.so at present. However, it seems that libuv for Linux arm is required to release .NET Core Runtime for Ubuntu ARM. Should I add libuv package for Linux arm? What do you think?

[moozzyk]

@jyoungyun - the only thing that currently requires libuv is the Asp.NET Core Http Server - Kestrel. If you don't use Asp.NET Core you can get by without libuv.
Currently we use kind of a homebrew way to build libuv for different platforms (one of the reasons was that their scripts did not support Windows ARM) but eventually it would be good just to build libuv using scripts from their repo which would make enabling new platforms easier.

[hqueue]

FYI. I've ran above dotnet tarball at Raspberry Pi3 which is ubuntu.16.04-arm.
At least, it works well with HelloWorld.exe and other simple applications. :) Thanks @jyoungyun !

[hqueue]

Do you have any automation around tests on real HW? My current thinking is to evaluate doing so in favor of emulator int he foreseeable future.

@gkhanna79 Unfortunately we don't have automated CoreCLR/CoreFX tests on HW and it also takes several hours to perform full CoreCLR unit tests on ARM device, e.g. Rpi2 or Rpi3.
We are also interested in the test infra but I'm afraid that full CoreCLR/CoreFX tests may not be feasible for ARM HW due to timeout, e.g. CoreFX tests run much longer than CoreCLR. Therefore I think we may have to choose tests even when the infra is available.

[stevedesmond-ca]

Looks like dotnet/core-setup#712 has been merged, so we can check that one off!

[gkhanna79]

Update build script to enable cross building (PR will be posted soon.)

@jyoungyun I believe you have completed this for Core-Setup, right? If so, can you please add PR details above and check this off :) ?

[gkhanna79]

Update build script to enable downloading and packaging for Linux/ARM.

@hqueue What is this item about (it is listed under Core-Setup section)?

[gkhanna79]

Should I add libuv package for Linux arm? What do you think?

@jyoungyun I think it will be a good idea to get the changes ready to build lib-uv for arm32 (as tracked by https://github.com/aspnet/libuv-build/issues/19).

[moozzyk]

@gkhanna79 @jyoungyun - is Linux/arm RID sufficient though? How about hard float vs. soft float? The linux-arm RID does not convey this information....

[stevedesmond-ca]

Convention in the dotnet repos seems to be that arm=armhf, and softp must
be specified explicitly. Perhaps an arm-softp rid could be added if
necessary

On Dec 27, 2016 7:01 PM, "Pawel Kadluczka" [e-mail] wrote:

@gkhanna79 https://github.com/gkhanna79 @jyoungyun
https://github.com/jyoungyun - is Linux/arm RID sufficient though? How
about hard float vs. soft float? The linux-arm RID does not convey this
information....

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/core-setup/issues/725#issuecomment-269400424,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAL9OFobHMUWCNgwxEvC8nLJMdvM1PJtks5rMabfgaJpZM4LD3ey
.

[hqueue]

I believe you have completed this for Core-Setup, right? If so, can you please add PR details above and check this off :) ?
What is this item about (it is listed under Core-Setup section)?

@gkhanna79 At first, we thought they were another tasks and I was planning to prepare seperate PR for each task. But as you already noticed, PR dotnet/core-setup#712 has enabled them altogether. I will checked them off. Thank you for notifying us :)

[hqueue]

is Linux/arm RID sufficient though? How about hard float vs. soft float? The linux-arm RID does not convey this information....

@moozzyk @stevedesmond-ca @gkhanna79 @jyoungyun I also have same question since when this task was started and for example there is no distinction in Ubuntun rids among different float ABI (e.g. hardfp, soft, softfp). This may affect all other dotent repos inclduing CoreCLR and CoreFX.

[jyoungyun]

Convention in the dotnet repos seems to be that arm=armhf, and softp must
be specified explicitly. Perhaps an arm-softp rid could be added if
necessary

@stevedesmond-ca @moozzyk @gkhanna79 @hqueue Linux RID does not contain the float ABI information, but we already use in combination with RID and float ABI information as a buildarch. As @stevedesmond-ca mentioned above, if necessary, we can add an arm-softfp rid like CoreCLR and CoreFX repos. I would like to keep the existing structure about RID and float ABI information in core-setup and libuv-build. And we are going to add arm-softfp to core-setup soon for supporting Tizen OS. Tizen only supports arm-softfp.

@gkhanna79 All of work for building core-setup for Ubuntu ARM are already merged via dotnet/core-setup#712. And I will post PR soon to disable GenerateDebs target regarding dotnet/core-setup#849.

[hqueue]

@gkhanna79 I'm also thinking about adding Tizen to Linux ARM32 and it will be started bottom up, i.e. starting from CoreCLR and CoreFX as we did it for ubuntu 14.04 and 16.04. I think we can follow similar steps. What do you think of it? I can build CoreCLR for Tizen right now and I can add rootfs for Tizen whenever approved.

[gkhanna79]

@hqueue @jyoungyun I must admit I am not very familiar with the difference between soft and hard fp. Can you please elaborate on them before we go down the path of determining whether they require new RIDs or not?

[gkhanna79]

adding Tizen to Linux ARM32 and it will be started bottom up

@hqueue In general, the concept sounds fine to me. however, before proceeding ahead with it, I would like to understand the difference between Tizen (built for Linux Arm32) and a regular Linux Arm32 .NET Core - can you share more details on it?

[hqueue]

Can you please elaborate on them before we go down the path of determining whether they require new RIDs or not?

@gkhanna79 GNU defines three floating ABIs for ARM, i.e. soft, softfp and hard. (see -mfloat-abi=name part at https://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html) The major difference between softfp and hard is as follows.

• GNU hard ABI: use of floating point instructions + use FPU-specific calling conventions (i.e. use FP registers for floating-point parameters and return values)
• GNU softfp ABI: use of floating point instructions + use soft-float calling conventions (i.e. use stack or integer registers for floating-point parameters and return values)

Therefore you can not use two ABIs together, because calling conventions are different.

In dotnet, we use hard and softfp for ARM. (no soft ABI.)

• hard ABI: a default build for ARM32 i.e. ./build.sh arm
• softfp ABI: we can build with ./build.sh arm-softfp

And all binaries in arm rootfs have hard ABI and binaries in arm-softfp rootfs have softfp ABI.

[hqueue]

I would like to understand the difference between Tizen (built for Linux Arm32) and a regular Linux Arm32 .NET Core - can you share more details on it?

@gkhanna79 Basically you can consider Tizen as another distribution of Linux/arm-softfp.
The major difference is version of libraries/kernel and composion of packages in each distribution, for example version of libicu is different.
The difference is similar to the difference between ubuntu.14.04 and ubuntu.16.04 in Linux Arm32 .NET Core.

In short,

• We have one ubuntu.14.04 rootfs for Linux/arm-softfp now and we can build for ubuntu.14.04 using ARM softfp ABI.
• If we add Tizen rootfs, we will have two rootfs for Linux/arm-softfp, i.e. ubuntu.14.04 and Tizen and we are able to build for ubuntu.14.04 and Tizen using ARM softfp ABI.

FYI. In CoreCLR, we have total 5 rootfs for Linux/arm (hardfp), i.e. jessie, vivid, trusty, sily and xenial.

I'm trying to explain in short to avoid confusion. If you have any question, let me know :)

[gkhanna79]

And all binaries in arm rootfs have hard ABI and binaries in arm-softfp rootfs have softfp ABI

Thanks for the explanation @hqueue. I concur that we lazily enable soft-fp if/once the need arises.

[gkhanna79]

Thanks for the brief explanation of Tizen @hqueue. Aside from the platform binaries/kernel version differences, is Tizen carrying the same set of binaries as .NET Core usually does? That is, when building a Tizen app, do you basically target .NET Core (and thus, have a shared FX based of Microsoft.NETCore.App) OR do you have (or plan to have) a different targeting mechanism (e.g. Samsung.Tizen package) that could carry a different set of binaries?

Also, how do Tizen apps get activated? Do they activate in the same manner as .NET Core apps (using dotnet.exe) or some other way?

[gkhanna79]

Basically, @hqueue, I am trying to understand if Tizen is just another distro equivalent for .NET Core or completely new targeting/AppModel experience.

[hqueue]

@gkhanna79 I thinks there are several questions here.

is Tizen carrying the same set of binaries as .NET Core usually does?

For CoreCLR and CoreFX, right. We will use exactly same set of binaries in Linux/arm-softfp, and AFAIK there is no difference between Linux/ARM and Linux/arm-softfp by now.

Tizen itself use RPM packaging system for platform. CoreCLR and CoreFX for Tizen are already being built with Tizen build system (https://build.tizen.org) at Tizen community (https://www.tizen.org).
CoreCLR for Tizen: https://build.tizen.org/package/show?package=coreclr&project=Tizen%3AMobile
CoreFX for Tizen: https://build.tizen.org/package/show?package=corefx&project=Tizen%3AMobile

That is, when building a Tizen app, do you basically target .NET Core (and thus, have a shared FX based of Microsoft.NETCore.App) OR do you have (or plan to have) a different targeting mechanism (e.g. Samsung.Tizen package) that could carry a different set of binaries?
Also, how do Tizen apps get activated? Do they activate in the same manner as .NET Core apps (using dotnet.exe) or some other way?

There are serveral app model in Tizen and .NET Core will be one of them. I think this can be explained by @lemmaa and @idkiller.
@lemmaa @idkiller Please look into this inquiries.

I am trying to understand if Tizen is just another distro equivalent for .NET Core or completely new targeting/AppModel experience.

@gkhanna79 I don't think it provide completely new experience. Because Tizen will act as another distro of Linux, at least I think you can consider it as another distro equivalent for .NET Core.
@lemmaa @idkiller what do you think of it?

I think we can continue the discussion next year for more detail :)

[jyoungyun]

@gkhanna79 @hqueue @hseok-oh What do you think of naming the arm-softfp as armsf for nupkgs? I think it does not look so pretty to have a - between buildarch name. Please give me good idea. :)

[hqueue]

@jyoungyun How about using armel for our arm softfp ? It seems that armel is used for ARM softfp binaries in Ubuntu and it would be great if we can follow conventions in Linux community.

See https://wiki.ubuntu.com/ToolChain

armel: soft-float ABI / hard-float ABI, defaulting to arm-linux-gnueabi.
armhf: hard-float ABI / soft-float ABI, defaulting to arm-linux-gnueabihf.

[idkiller]

@gkhanna79 If target means "base framework library", current Tizen use Microsoft.NETCore.App as a base framework for developing User apps.
but we only use it as a "base framework", Tizen Apps does not executed like other dotnet application.
Tizen has not used dotnet.exe or another tool like corerun, coreconsole.
Tizen has own original launcher and internally it use interface of libcoreclr.so directly.
In this model, binary and dlls in coreclr and Microsoft.NETCore.App should pre-installed to device and app should be run like Framework-dependent deployments. actual App package only has dll or exe of app or app's dependency.
but these days, we prepare another model like Self-contained deployments. it should use in device that does not has a runtime.
As for the Packaging, Tizen apps are not packaged with nupkg. (it use tpk)
there are some APIs for Tizen apps (https://git.tizen.org/cgit/?q=csapi)
internal build for these APIs in GBS and build for user app in VisualStudio use nupkg but app is not packed to nupkg.

[gkhanna79]

@jyoungyun I concur with @hqueue on following the convention. Given that we already have an "arm" RID (at https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json), we will use it for hardfp (and synonym for armhf above). For softfp support, armel sounds good to me and something that we should add at https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.json.

CC @janvorli

[janvorli]

It makes sense to me.

[jyoungyun]

@hqueue @gkhanna79 @janvorli I also agree with @hqueue opinion! Sounds good to me too. I will post a PR to support armel build on core-setup soon.

[hqueue]

@gkhanna79 @janvorli Cool! We prepared PR for armel in CoreCLR and CoreFX and updated the agenda of this issue releated to armel. We are working on Core-Setup too. Also I have updated Tizen topic in arm-softfp (armel) part. Please also take a look :)

[hqueue]

We have one ubuntu.14.04 rootfs for Linux/arm-softfp now and we can build for ubuntu.14.04 using ARM softfp ABI.

@gkhanna79 I've made a mistake in previous answers. For ARM softfp, we use debian 8 (jessie) rootfs not ubuntu14.04. Therefore rid will be debian.8-armel if we use armel for ARM softfp.

[gkhanna79]

@hqueue i have cleanedup/reordered the workitems for armel and Tizen.

[moozzyk]

@gkhanna79 https://dotnet.myget.org/feed/aspnetcore-ci-dev/package/nuget/Libuv/1.10.0-preview1-22036 contains arm versions of Libuv for Linux. Let me know once the package has been replicated and I can update https://github.com/dotnet/core-setup/blob/master/pkg/projects/Microsoft.NETCore.App/project.json#L43

[gkhanna79]

@moozzyk I have uploaded it to myget and should be visible as soon as myget will make it visible in :)

[moozzyk]

@gkhanna79 - updated. I guess you can mark "Build LibUV for Arm32 (Issue aspnet/libuv-build#19)." checked.

[gkhanna79]

@moozzyk Done!

[jyoungyun]

@gkhanna79 Should we provide sdk to release arm and armel officially? And.. should the sdk tarball support to resolve the NETCoreApp1.1(it will be NETCoreApp2.0) for the official release? Can you tell us what we need to prepare for an official release? At present, the sdk tarball what I made from cli project does not resolve NETCoreApp1.1 due to Unable to resolve dependency 'Microsoft.CodeAnalysis.CSharp' error. So I am wondering if I have to solve this problem for the official release and if I have to fix roslyn to fix it.

[Ramon-Balaguer]

Please @jyoungyun , can you upload your HellowWorld project and the bins? I have:
Raspberry Pi3 + ubuntu.16.04-arm + dotnet-ubuntu-arm.1.2.0-beta-001206-00.tar.gz.

Could you please tell me how to compile a netcore project to run on Raspberry? I'm using windows.

Thanks

[jyoungyun]

@scrambler86 Hi, I attached sdk tarball for ubuntu.14.04-arm. Even this file is for ubuntu.14.04-arm but i think it's okay to run on ubuntu.16.04-arm. After installing sdk tarball, you can use dotnet new command for creating a HelloWorld project. I usually build core-setup and cli project in the following order.

1. Build CoreCLR for ubuntu.14.04-arm
2. Build CoreFX for ubuntu.14.04-arm
3. Copy result nupkgs of CoreCLR and CoreFX to specific directory and add below line to NuGet.Config file in core-setup.

<add key="Local" value="The speicific directory path" />

1. Build Core-setup with ./build.sh --skiptests --env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.14.04-arm,CROSS=1,ROOTFS_DIR=/home/jyoung/git/dotnet/rootfs/arm command.
2. Copy results nupkgs of core-setup to specific directory and add upper line to NuGet.Config in cli.
3. Build cli with ./build.sh --env-vars DOTNET_RUNTIME_ID=ubuntu.14.04-arm -c Release /p:CLIBUILD_SKIP_TESTS=true /p:DISABLE_CROSSGEN=1 command.
   But to use --env-vars option, cli needs https://github.com/dotnet/cli/pull/5290 PR.

dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz

[Gonkers]

@jyoungyun Thank you for your help it's much appreciated!
@scrambler86 I just found this post as well that could be helpful https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md

[jyoungyun]

I updated the shared runtime tarballs for ubuntu.14.04 arm and ubuntu.16.04 arm. These tarballs contain libuv.so and CodeAnalysis dlls that were missing from the last tarballs.

And I also uploaded the sdk tarballs for rpi3.
You can use it in the following order.

1. Extract the matching shared runtime and sdk tar.gz to the same folder on your Pi and go into that folder in the terminal.
   dotnet-ubuntu.14.04-arm.1.2.0-beta-001291-00.tar.gz (shared runtime)
   dotnet-sdk-ubuntu.14.04-arm.1.0.0-preview5-004431.tar.gz (sdk)
   dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz (shared runtime)
   dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz (sdk)

mkdir dotnet
tar xvzf dotnet-ubuntu.16.04-arm.1.2.0-beta-001291-00.tar.gz -C ./dotnet
tar xvzf dotnet-sdk-ubuntu.16.04-arm.1.0.0-preview5-004431.tar.gz -C ./dotnet

1. Make a symbolic link from dotnet to /usr/bin/dotnet.
   ln -sf /usr/bin/dotnet /yourpath/dotnet/dotnet
2. Create a folder named helloworld and go into it.
3. Run dotnet new then dotnet restore then dotnet publish to publish your helloworld app.
   But, in my case, I couldn't success for resolving NETCoreApp1.1 yet even though I have the related nupkg files.

[Petermarcu]

I just pushed some simple instructions to dotnet/core master. They point at some older tarballs. I will adjust it to point to these. Looking forward to when these can be part of the daily build. :)

https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md

[Petermarcu]

@scrambler86 , in the next couple weeks, we should be able to get all the packages published in order to actually build a standalone app on Windows that that targets arm32 linux. This is what we need to have the developer experience on Windows targeting Raspberry Pi complete. For now, if the app you write doesn't carry the runtime with it and can just run on the shared runtime, then you should be able to publish on Windows and copy the published output to your Pi. Give it a try and let us know.

[Petermarcu]

@jyoungyun, do you have a nuget.config that points to a local folder containing your packages? Without that, I dont think it would know how to find them.

[gkhanna79]

@jyoungyun All master branches will be updated to resolve .NETCoreApp 2.0. However, before that happens, we need to have the nuget packages published that I am working through since the build-rootfs.sh fails in our official build infrastructure.

[JensNordenbro]

So all this nice work is aiming for 2.0, or is it 1.2 release?

[stevedesmond-ca]

1.2 is in the process of being renamed 2.0

[hqueue]

However, before that happens, we need to have the nuget packages published that I am working through since the build-rootfs.sh fails in our official build infrastructure.

@gkhanna79 Do you have problem with buiild-rootfs.sh for arm and armel, i.e. ./build-rootfs.sh arm and ./build-rootfs.sh armel? What kind of problem are you suffering ? I think we can also look into the porblem if possible.

[Petermarcu]

It was something about building in Docker which is how we build many of our Linux legs of our build.

[jyoungyun]

@Petermarcu Yes, I updated a NuGet.Config file to point to a local folder containing our nupkgs. But it was still failed to resolve NETCoreApp1.1. The error message is like below.

Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/system.globalization/index.json'.
  A task was canceled.

I doubt if it is an issue of the corporate network. Have you ever seen such an error?

[Petermarcu]

I haven't. @ericstj any thoughts?

[ericstj]

NuGet will usually fail if it has any feed in its list that can't be resolved. NuGet/dotnet restore will usually echo all the feeds its using as well as all the config sources to the console when you run it. Double check that you aren't using any nuget feeds that are unresolvable in your private network.

[JensNordenbro]

Ok, 2.0 it is. Thanks @stevedesmond-ca.
So the the core-setup effort for arm will debute at 2.0 release?

[hqueue]

It was something about building in Docker which is how we build many of our Linux legs of our build.

@Petermarcu @gkhanna79 FYI. Among arm rootfs, rootfs for armel(Tizen) (constructed from build-rootfs.sh armel tizen) has more simpler structure than other rootfs (ubuntu and debian) and requires less effort. For example, armel(Tizen) rootfs just contains only required components for build (about 100MB), but other rootfs, i.e. arm (Ubuntu) and armel (Debian), have all Linux components of default Linux distro (about 700MB) and requires debootstrap operation with sudo previledge during build-rootfs.sh. Therefore requires more resources including network bandwidth, storage and computation.
We also experience failure sometime in host x64 Linux box when constructing rootfs for ubuntu-arm and debian-armel due to various reasons including network problem and etc. Therefore if you are experiencing trouble to construct ARM rootfs in docker environment, I would suggest trying simpler Tizen rootfs.

[gkhanna79]

@hqueue The fact that Docker images are a container and not a virtualized environment created problems - it had nothing to do with the rootfs creation.

The specific issue was that I was able to build LinuxArm32 in Docker image on my dev machine but couldn't do so in our official build. After investigating various options, I took a different approach to compare differences between the success and failing case. The issue is that, even when attempted within a Docker container, the cross build expects (in the second stage invocation that qemu-debootstrap kicks off) that the host machine has the cross build dependencies also installed. Since they were not present in the host OS image in our official build, and only present in the Docker container, cross build within Docker container would fail.

I took one of our official build machines offline yesterday, got the missing packages installed and got a clean Linux Arm32 build in a Docker image finally! :) I will get our official machines updated and will soon get this going.

With the above said, we need to produce .NET Core for both armhf and armel. Thus, debian.8-armel or Tizen armel build will be an additive and not alternative to armhf build.

[hqueue]

@gkhanna79 Glad to hear the progress and thank you for all your efforts :)

[gkhanna79]

Updating the primary thread - CoreCLR pipeline build is now publishing Arm32 (armhf) packages for Ubuntu 14.04 and Ubuntu 16.04 on myget! Here are the first set of packages that got published today from the repo:

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.14.04-arm.Microsoft.NETCore.Runtime.CoreCLR

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.16.04-arm.Microsoft.NETCore.Runtime.CoreCLR

[hqueue]

@gkhanna79 First of all great thanks for publishing packages for CoreCLR :)

We tested with above packages, but unfortunately, I think it has bug due to clang which is used when building CoreCLR.

Current llvm-3.8 and below can not build CoreCLR and CoreFX correctly with -O3 optimization for ARM, because they have bug related to TLS. You can find details at https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md#how-to-enable--o3-optimization-level-for-armlinux
(There are lots of issues related to this. One of them is https://github.com/dotnet/coreclr/issues/6530 )

Therefore I think we have to consider upgrading llvm version for ARM release and we will also discuss this issue with issue dotnet/core-setup#790 (arm32 CI). Let's move to dotnet/core-setup#790 and I will continue discussion there.

BTW Most simple workaround is using "-O1" instead of "-O3" when building CoreCLR with current infra structures.

[gkhanna79]

Very well @hqueue - I will continue the discussion with you on dotnet/core-setup#790.

[stevedesmond-ca]

Heads up, in case anyone else wants them, I've got automated builds of the .NET Core Runtime for ubuntu.16.04-arm working now, via a bunch of bash scripts.

https://github.com/stevedesmond-ca/dotnet-arm/releases/

I'm planning on publishing nightlies (including the SDK within the next week) there until official CI builds can be set up.

[gkhanna79]

Excellent @stevedesmond-ca! I am working on the CoreFX support for official builds right now.

BTW, are your builds for Ubuntu 14.04 or 16.04? And I take it you are consuming the packages we publish for CoreCLR already, right?

[stevedesmond-ca]

16.04 only right now. I'm building CoreCLR and CoreFX from scratch, then feeding artifacts from those into core-setup, very similar to how @jyoungyun described here.

Once CoreCLR on myget is built with -O1 (currently I get the -O3 TLS segfault from what's pulled down) it should be easier to use official builds. I do think there's value in being able to do the whole thing offline, to decouple the build process from the current CI publish process, but that's just my personal preference.

[bjoernbusch]

@stevedesmond-ca I tried running your built in a docker container on my RPi3 with the following dockerfile:

FROM armv7/armhf-ubuntu:16.04

ENV VERSION 2.0.0-beta-001455-00

RUN apt-get update -q -y && \
apt-get upgrade -q -y && \
apt-get install -q -y wget

RUN wget https://github.com/stevedesmond-ca/dotnet-arm/releases/download/$VERSION/dotnet-ubuntu-arm.$VERSION.tar.gz && \
mkdir -p /usr/share/dotnet && \
tar -xf dotnet-ubuntu-arm.$VERSION.tar.gz -C /usr/share/dotnet/ && \
ln -sf /usr/share/dotnet/dotnet /usr/bin/dotnet

RUN mkdir dotnet
COPY dotnet-arm/hello-world/hello-world.csproj dotnet/hello-world.csproj
COPY dotnet-arm/hello-world/hello-world.deps.json dotnet/hello-world.deps.json
COPY dotnet-arm/hello-world/hello-world.dll dotnet/hello-world.dll
COPY dotnet-arm/hello-world/hello-world.pdb dotnet/hello-world.pdb
COPY dotnet-arm/hello-world/hello-world.runtimeconfig.json dotnet/hello-world.runtimeconfig.json
COPY dotnet-arm/hello-world/Program.cs dotnet/Program.cs

I then start the container with docker run -it core /bin/bash and execute in dotnet folder dotnet hello-world.dll, but then I get an error:

Failed to load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0-beta-001455-00/libcoreclr.so, error: libunwind.so.8: cannot open shared object file: No such file or directory
Failed to bind to CoreCLR at '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0-beta-001455-00/libcoreclr.so'

Any hint on what stupid mistake I'm making?

[stevedesmond-ca]

@bjoernbusch check out https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md -- specifically the "prereq packages" part -- chances are those aren't in the base image

[bjoernbusch]

Yes! facepalm that was it, now it's working! Thanks!

[jyoungyun]

@stevedesmond-ca Thank you for sharing your tarball :) It works well on my raspberry PI3 too. Now I'm trying to build core-setup using docker. Does your CI system use docker for releasing, doesn't it? But when I tried to build with docker, I was faced with a little problem that all arguments that match the docker argument were removed by https://github.com/dotnet/core-setup/blob/master/build.sh#L36L37 lines. Have you ever built core-setup for Ubuntu.16.04-arm in docker? My build command line is like below. After parsing on build.sh, some arguments are removed.

./build.sh --docker ubuntu.16.04 --env-vars "DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=ubuntu.16.04-arm,CROSS=1,ROOTFS_DIR=/opt/code/cross/rootfs/arm"
->
Argumtens~~~~:--env-vars DISABLE_CROSSGEN=1,TARGETPLATFORM=arm,TARGETRID=-arm,CROSS=1,ROOTFS_DIR=/opt/code/cross/rootfs/arm

I will make a PR soon. If I'm wrong, let me know.

[stevedesmond-ca]

I'm just doing my builds from a xenial VM, no Docker involved.

[ThadHouse]

Thanks for sharing the debian softfp tarball! I was able to get it working on one of the embedded platforms I normally work with (The NI roboRIO for FRC). Worked fantastically, and even was able to run a fairly complex socket based program with no issues I could find. Going to be so nice not to have to custom compile Mono anymore.

[hqueue]

Worked fantastically, and even was able to run a fairly complex socket based program with no issues I could find

@ThadHouse Glad to hear it helps. And also goot to hear that real complex applications run on dotnet for debian (softfp) in real device :)

[jyoungyun]

@ThadHouse Actually I tested the operation for debian.8-armel tarball on DockerImage by using rpi3. It is really good news that it works well on real device. :)

[jyoungyun]

@gkhanna79 Could you let me know the progress of nupkg uploading for arm in core-setup? AFAIK this should be preceded, CoreFX nupkgs can upload to MyGet Feed. Thank you for your continued support.

[gkhanna79]

@jyoungyun I have done trial builds over the weekend and they look good and expect to have them running by Tuesday if all goes well. I will keep this discussion updated with the status.

[ThadHouse]

If you get the NuGet packages, I will be happy to help test the debian.8 version.

Note I am not actually testing on a debian system, but its actually an arch system. But it does have the same sysroot, or a very close one from what I can tell.

[jyoungyun]

@gkhanna79 My mistake :( Uploading CoreFX nupkgs should be ahead of core-setup job. And I found the CoreFX nupkgs for ARM in dotnet-core myget. Thank you.

[gkhanna79]

We now have Ubuntu 14.04 and 16.04 Arm32 builds getting published daily! Here is the first set of packages that came out today:

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.14.04-arm.Microsoft.NETCore.App/2.0.0-beta-001559-00

https://dotnet.myget.org/feed/dotnet-core/package/nuget/runtime.ubuntu.16.04-arm.Microsoft.NETCore.App/2.0.0-beta-001559-00

[tmds]

We now have Ubuntu 14.04 and 16.04 Arm32 builds getting published daily! Here is the first set of packages that came out today:

This is great! Are there plans for a portable arm32 build? It would be super to have dotnet run on
Rasbian, Fedora, ... too.

[gkhanna79]

@tmds Yes, we plan to look into that soon.

[TheRadziu]

is there any build I can use on Raspbian Pixel now ? Today I updated my distro which somehow broke mono I was using up to this point. If not is there any ETA on builds that can be used on raspberry pi?

[hqueue]

@TheRadziu I'm not sure whether it works on raspbian pixel correctly or not but https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu.16.04-arm.latest.tar.gz is for ubuntu-mate (ubuntu.16.04-arm) and it works well on my rpi3 with ubuntu-mate.
Both raspbian and ubuntu-mate are based on debian, so I think you can try it for raspbian too. If there is version issue, ubuntu.14.04-arm is also available at https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu-arm.latest.tar.gz.

You can find them in daily build page at https://github.com/dotnet/core-setup/.

[TheRadziu]

@hqueue thanks for your answer! Sadly, I cant make it work
./dotnet: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: versionGLIBCXX_3.4.21' not found (required by ./dotnet) when I checked it bystrings /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 | grep GLIBCXX`it listed GLIBCXX_3.4 up to GLIBCXX_3.4.20.
I wonder if I can update it somehow

[hqueue]

@TheRadziu Please try ubuntu.14.04-arm at https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu-arm.latest.tar.gz

It seems that ubuntu.14.04-arm requires 3.4 and 3.4.18 not 3.4.21.

[tmds]

@TheRadziu @hqueue fyi, glibc has a backwards compatibility feature. The portable linux x64 build of dotnet requires glibc 2.14 as a minimum. The portable arm32 build will have a similar requirement.

[hqueue]

@gkhanna79 I found out that dotnet/core-setup#789 (Enable Pipeline Build support for Linux Arm32) is closed recently for arm.
Can we add "Enable Pipeline build support for Linux Arm32" for armel(tizen) and make a new issue for the item ? Let us know your plan for armel pipeline, so we can keep pace with you :)

[gkhanna79]

@hqueue Yes, please create a new issue (assigned to me) for Tizen pipeline build for each of the 3 repos and add it to the workitem list for Tizen above.

[jyoungyun]

@gkhanna79 What do you have a 2.0.0 release plan about sdk ? Currently I can't find branch for 2.0.0 release in cli.

[Petermarcu]

@jyoungyun , take a look at the master branch of the cli repo.

[gkhanna79]

@hqueue @hseok-oh @jyoungyun Over the last week, I have switched our official pipeline builds to use the _ubuntu1404_cross_prereqs_v2_ and _ubuntu1604_cross_prereqs_v2_ Docker images that contain the arm32 toolset in _/crossrootfs/arm_. Can you please look into update the CI scripts you are adding/added for CoreCLR/CoreFX/Core-Setup repos to use them and remove the build-rootfs step?

[hqueue]

@gkhanna79 Cool! We will update the script.

[jyoungyun]

@Petermarcu Thank you!

[challe]

I'm trying to create a .NET Core Web API for my Raspberry Pi 3 running Ubuntu 16.04 with the lates 2x runtime. I'm using Visual Studio Code on my Windows when developing. My current SDK version is 2.0.0-alpha-004853, but when I do dotnet new webapi it adds these package references in my csproj-file:

<PackageReference Include="Microsoft.AspNetCore" Version="1.0.3" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="1.0.2" />
<PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="1.0.1" />

I have heard this is the reason I can't make it run when I then try to publish the file using dotnet publish --runtime ubuntu.16.04-arm. Perhaps there are other issues I might encounter as well.

I'm fairly new to all of this, so I'm not sure what my next step is. How do I include the 2x compatible ones? Is what I'm trying to do even possible at the moment?

[gkhanna79]

@challe Please file an issue in CLI repo to drive the discussion there about how to get a CLI SDK that supports 2.0 framework (which is under development right now).

[challe]

I'll do that. Thanks.

[rahulreddy65]

Hi, I tried to run 1.0.0-preview5-004431 version but it doesn't work. It doesn't even work with hello world. Can anyone please tell me if there is any working version which I can use? If you can, please post the download link. Thank you.

[gkhanna79]

@rahulreddy65 Lets discuss this in https://github.com/dotnet/cli/issues/5868.

[hseok-oh]

@gkhanna79 Please apply two items in docker images in your next docker images update.

• Add x86 rootfs and toolchain in ubuntu1404 or/and 1604 (and new x86 docker image for test). We are preparing x86 linux CI build.
• Apply dotnet/coreclr#9897 in ubuntu1404 rootfs to build libcoreclrtraceptprovider.so

[hseok-oh]

I makes new issue for x86 docker image. dotnet/coreclr#9903.

[jyoungyun]

@gkhanna79 Can I know when the tizen nupkgs are upload to MyGet server? I wanna enable tizen CI in core-setup. And... are you going to upload ubuntu arm tarball to Azure server? I'm trying to build cli for ubuntu arm but it depends on Azure server to download runtime tarball.

[gkhanna79]

Ubuntu tarball is available - see https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu.16.04-arm.latest.tar.gz. Does this not help @jyoungyun ?

[jyoungyun]

Oh, I see. I can use it. The same tarball is on both paths below. Thank you.
https://dotnetcli.blob.core.windows.net/dotnet/master
https://dotnetcli.azureedge.net/dotnet/master

[jyoungyun]

@gkhanna79 Do you want to enable ubuntu 16.04 arm CI by default ?

[gkhanna79]

@jyoungyun Is there significant difference between the 14.04 and 16.04? IMHO, unless we start to see issues unique to it (which will manifest as build breaks in official pipeline), having one should be sufficient. What do you think?

[olegsavelos]

@hqueue Am i correct in my assumption that this build dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz should run on PI Zero Raspbian?
I have tried it but get segmentation fault when executing dotnet command. Anything i can do to resolve it?

[Petermarcu]

I don't think it works on Pi Zero. I don't think anyone has looked at why.

[ThadHouse]

@olegsavelos the debian.8-armel build will not work on a raspberry pi zero. All of the builds here are for armv7 devices, with the different versions selecting between hard float ABI and soft float ABI. The Pi Zero (and the original Pi) is an armv6 device. I do not know what it will take to get armv6 working and building, but it could range from fairly easy (changing compiler flags) to very difficult. I just don't know.

[olegsavelos]

Can anyone shed some light on Armv6 support ?

[JensNordenbro]

@jyoungyun , will the .net core tizen build work on an image including https://github.com/TizenTeam/meta-tizen on yocto?

[hqueue]

@olegsavelos If you meant PI Zero with Broadcom BCM2835, I'm afraid it's not easy to work dotnet-debian.8-armel.1.2.0-beta-001271-00.tar.gz on your Pi Zero. I also agree with @ThadHouse that PI Zero is based on armv6 and it could be range from easy to very difficult.

AFAIK current dotnet for ARM make use of Thumb-2 ISA which is not available in armv6 and vfp feature which is mostly available in armv7, but only optinally available in armv6. Armv6 support old Thumb not Thumb2, therefore you may have to build (1) all native component of dotnet for armv6 and also (2) may have to update JIT compiler and sub-component(written in assmebly language) of CoreCLR and CoreFX to make use of Thumb ISA and supported vfp feature of Broadcom BCM2835 instead of armv7. But I can't estimate how much effeort will be needed.

[olegsavelos]

Mono seems to be working fine on armv6 so wouldn't it be possible to port the portions of code related to armv6 to .net core? In any case i think PI Zero is excellent platform for wide range of solutions and it would be shame not to support it.

[rahulreddy65]

Can anyone tell me when will I be able to build and run programs on arm32 linux? thanks

[hqueue]

@rahulreddy65 If you are using Ubuntu 14.04 or Ubuntu 16.04 for arm, I think you can run C# programs on arm32 linux by installing binary from https://github.com/dotnet/core-setup#daily-builds.

• Ubuntu 14.04-arm https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu-arm.latest.tar.gz
• Ubuntu 16.04-arm https://dotnetcli.blob.core.windows.net/dotnet/master/Binaries/Latest/dotnet-ubuntu.16.04-arm.latest.tar.gz

If you want to build C# program from arm32 linux, it's not available yet. However you can build C# program from other environment, e.g. x64 linux or x64 Windows, and run the program on Ubuntu arm.

And there is another way you can just build and run C# program for arm32 linux with runtime framework version 2.0.0-beta-001620-00 and later. Please take a look at https://github.com/dotnet/core/blob/master/samples/RaspberryPiInstructions.md
However it also requires x64 environment to build the C# program.

[hqueue]

@olegsavelos I think you'd better open an issue about supporting armv6 at https://github.com/dotnet/coreclr, because most of required works are relevant to coreclr and experts who can answer your questions(e.g. supporting new CPU) are there too :)

[olegsavelos]

@hqueue Thanks ! Will do that :)

[jyoungyun]

Finally, I got below results!!!

pi3@raspberry:~/Downloads/c$ time NUGET_PACKAGES=/home/pi3/Downloads/c/p dotnet build
Microsoft (R) Build Engine version 15.2.47.30403
Copyright (C) Microsoft Corporation. All rights reserved.

  c -> /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:27.08

real    0m40.917s
user    0m43.890s
sys 0m1.390s

pi3@raspberry:~/Downloads/c$ time NUGET_PACKAGES=/home/pi3/Downloads/c/p dotnet run
Hello World!

real    0m51.861s
user    0m54.380s
sys 0m1.870s

The performance is very.... low, but Raspberry PI can build cs file itself!

[janvorli]

@jyoungyun are these results from release build of coreclr?

[jyoungyun]

@janvorli All of the nupkg used in the cli build got downloaded from myget server so this runtime(coreclr) is release build with -O1 option. If clang3.9 is enable, the result will be faster. :)

[janvorli]

@jyoungyun it was discovered in another issue that dotnet run command adds quite a lot of overhead to the start of the managed app (on x64 Linux, it adds about 2 seconds) compared to passing the managed assembly directly to dotnet without the "run".
Could you please try to measure that case as well? Just to be clear, I mean running time dotnet /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll

[jyoungyun]

@janvorli In raspberry PI3, passing dll as a directy argument is about 40 seconds faster than using dotnet run. This seems to be quite a lot of overhead in dotnet run command...

pi3@raspberry:~/Downloads/c$ time dotnet run
Hello World!

real    0m44.373s
user    0m56.000s
sys 0m1.800s

pi3@raspberry:~/Downloads/c$ time dotnet /home/pi3/Downloads/c/bin/Debug/netcoreapp2.0/c.dll
Hello World!

real    0m1.389s
user    0m1.340s
sys 0m0.040s

[janvorli]

@jyoungyun Thank you for measuring it! We really need to do something with it. I'll add this information to the other issue where the slowness was reported for x64.

[hqueue]

@janvorli @jyoungyun I think one of huge timz difference can be due to lack of native image for arm cli, which may be available for x64 cli itself.

[janvorli]

Well, even on x64, there is a huge difference in those two ways of launching the app. See https://github.com/dotnet/cli/issues/6241

[nzain]

I see a lot of progress related to the Raspberry PI 2/3 and Ubuntu... but what is required to get my dotnet core 2 project running on an i.MX7 (Cortex-A7) with an angstrom based linux?

From my limited understanding the architecture should be armhf, which is the default architecture in all dotnet cross-compile guides I found so far.

I started cross-compiling CLR and FX, but it all seems centered around ubuntu. Will the result work on angstrom, as long as the required prerequisites (like libunwind, etc) are installed?

[JensNordenbro]

@nzain - try running using Linux (armhf) (for glibc based OS) in table on core-setup-readme.
This works fine for us (imx6 yocto poky) however you might have to bring in native dependencies into your os -dist manually.
Also notice that dotnet/core-setup#2358 is not completely resolved however code is merged so it might just be that the fixes are not validated.

[zeroskyx]

.NET Core 2.0 works flawlessly on a Raspberry Pi 3 with Raspbian Jessy Lite. The following is required to get it working:

On the vanilla installation of the OS run: sudo apt install libunwind8

In your .NET Core project include:

<TargetFramework>netcoreapp2.0</TargetFramework>
<RuntimeIdentifiers>win10-x64;linux-arm</RuntimeIdentifiers>

To publish on the command line/ vscode:

dotnet restore
dotnet build
dotnet publish --output Publish --runtime linux-arm

When your project name is MyProject: copy all the files in the Publish directory to the Pi and run

chmod 744 MyProject
./MyProject

Enjoy :)

[gkhanna79]

@hqueue @jyoungyun Can we close this issue since the outstanding issue (per the list above) is in CoreCLR repo and tracked by https://github.com/dotnet/coreclr/issues/8549?

[hqueue]

@gkhanna79 Sure. I will mark https://github.com/dotnet/coreclr/issues/8549 in the list as done with comment.

[hqueue]

Closing this issue and remaining CoreCLR issues will be discussed at https://github.com/dotnet/coreclr/issues/8549