Runtime: How get and build test version of dotnet ARM.

[tomaszchacuk]

Hi.

I am intending to create some piece of software that would target ARM Linux (Ubuntu 16.04 + Rasppberry PI 2/3). Due to the fact that dotnet core for arm is planned to be released in unknown time I would like to take whatever there is to run dontet core on arm. I require it for Kestrel server (Websockets, and HTTP, but not full ASP.MVC)

IWhat is the branch location for the arm version and how can I build it on my own ? I need to have "something" and cannot wait with starting the project till the end of the year .

[pgavlin]

cc @richlander @cmckinsey @gkhanna79

[mkborg]

@tomaszchacuk
dotnet for ARM is in proof of concept quality state. In fact it is completely broken. ;-)
If you want to program in C# just forget about dotnet and use mono (http://www.mono-project.com).
mono is already available in Debian (and Ubuntu) for ARM.
mono also has poor quality but at least sometimes it works. ;-)

[jkotas]

What is the branch location for the arm version

All arm specific changes are in master in this repo.

How can I build it on my own ?

The instructions to build CoreCLR are at https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md. However, this will build just the coreclr runtime, not the dotnet tool - that I believe does not work on ARM well yet. You can try to make this work by using dotnet tool on x64, and copying the build artifacts over to your ARM machine.

@leemgs Have you or somebody on your team tried running ASP.NET on ARM?

[tomaszchacuk]

If you want to program in C# just forget about dotnet and use mono (http://www.mono-project.com).

Oh believe me I tried - this was the first thing. The problem is I don't want to use ASP .NET 4 but ASP.NET Core(kestrel) wich works on windows but compiled when moved to linux does not work on mono (451 profile) Simple Hello world without Kestrel done the same way works. There are issues arround the net about this.

I am at the point where just nothing works properly :-)

[mkborg]

Friend of mine had issues with network sockets in latest mono version. We solved it reverting mono to one of previous release versions (sorry I don't remember exact numbers).

[leemgs]

@leemgs Have you or somebody on your team tried running ASP.NET on ARM?

Still Not. Most of the members concentrate on .NET Core's stabilization (both CoreCLR and CoreFX unit-tests) after enabling .NET Core on Linux/ARM32.

[leemgs]

I am intending to create some piece of software that would target ARM Linux (Ubuntu 16.04 + Rasppb....

@tomaszchacuk , If you try to build/run .NET Core (CoreCLR + CoreFX) for Raspberry Pi2 board + Ubuntu/ARM 14.04 32bit (hardfp), I recommend that you use Ubuntu 14.04 LTS X64 to avoid unnecessary engineering cost.

[gkhanna79]

.NET Core repos (CoreCLR, CoreFX, Core-Setup, CLI) are not yet enabled to build for Arm32 (on either Windows or Linux) in the open.

I need to have "something" and cannot wait with starting the project till the end of the year .

Enabling for Arm32 would mean to support building and testing, at minimum, the above 4 repos. If you must start immediately on this, the right thing would be to start enabling the various repos and bring them up. We can guide you in the process - and CoreCLR would be a good starting point.

What do you think?

[manu-st]

@mkborg We ported our tooling (3D Game engine) using CoreCLR on ARM and it works quite nicely. I would not say it is completely broken although it is still in the early stages.

[leemgs]

We ported our tooling (3D Game engine) using CoreCLR on ARM and it works quite nicely

@manu-silicon , Yes You mentioned your status previously. :+1: Can you share us the demo (e.g. Youtube video or others) of 3D game engine that you have successfully ported on ARM Tegra board?

[parjong]

@tomaszchacuk @jkotas I just tried the following 'Hello, World' example:

using System;
using System.IO;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;

namespace AspDotnetExample
{
  public class Runner
  {
    public static void Main(string[] args)
    {
      var host = new WebHostBuilder()
        .UseKestrel()
        .UseStartup<Service>()
        .Build();

      host.Run();
    }
  }

  public class Service
  {
    public void Configure(IApplicationBuilder app)
    {
      app.Run(async (context) => {
          await context.Response.WriteAsync(
              "Hello World. The Time is: " +
              DateTime.Now.ToString("hh:mm:ss tt"));

      });
    }
  }
}

I first build and publish the above example using dotnet from Ubuntu 14.04 (x64) where dotnet is available, and copy the published binaries into ARM device where we run CoreCLR unittest. Then, I just tried to run the example with corerun included in the testing binaries (CoreCLR + CoreFX):

Unfortunately, I just encountered the following error:

$ ./corerun ../hello_world.dll

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to load DLL 'libuv': The specified module could not be found.
 (Exception from HRESULT: 0x8007007E)) ---> System.DllNotFoundException: Unable to load DLL 'libuv': The specified module could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.NativeMethods.uv_loop_size()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.loop_size()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvLoopHandle.Init(Libuv uv)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object parameter)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Start(Int32 count)
   at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Start[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Start()
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host, CancellationToken token, String shutdownMessage)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
   at AspDotnetExample.Runner.Main(String[] args)

It seems that libuv is necessary, but currently it is missing in the testing environment.

@tomaszchacuk You may try this approach on your RPi. This approach requires only CoreCLR and CoreFX (CLI and core-setup are not required). Currently, it is possible to cross-build both CoreCLR and CoreFX for ARM/Linux although those are at the very early stage.

[leemgs]

(Unable to load DLL 'libuv': The specified module could not be found.
It seems that libuv is necessary, but currently it is missing in the testing environment.

If the reason is only the lack of the libuv library, I recommend that @tomaszchacuk install libuv library manually on Raspberry Pi board as following:

rpi2@arm# sudo apt-get install gyp
rpi2@arm# wget http://dist.libuv.org/dist/v1.0.0-rc1/libuv-v1.0.0-rc1.tar.gz 
(The latest version is v1.9.1: http://dist.libuv.org/dist/v1.9.1/libuv-v1.9.1.tar.gz)
rpi2@arm# tar -xvf libuv-v1.0.0-rc1.tar.gz
rpi2@arm# cd libuv-v1.0.0-rc1/
rpi2@arm# ./gyp_uv.py -f make -Duv_library=shared_library
rpi2@arm# make -C out
rpi2@arm# sudo cp out/Debug/lib.target/libuv.so /usr/lib/libuv.so.1.0.0-rc1
rpi2@arm# sudo ln -s libuv.so.1.0.0-rc1 /usr/lib/libuv.so.1

Then run "rpi2@arm# k kestrel" command, 
Started

[manu-st]

@leemgs I do not have a video handy, but what we were able to run on the board is what we made compile on Linux x64 too. See the tweet we made about our Linux support (https://twitter.com/xenko3d/status/746146402850340864)

[tomaszchacuk]

@parjong I tried this approach. Installed libuv. The next problem was missing W2_32.dll.

I started trying to build corefx and corecls. CoreClr succeeded, but corefx. I will post the errors as soon as I return home

[jkotas]

The next problem was missing W2_32.dll.

This means that you are using Windows-specific build of some library. Do you happen to have callstack of the failure?

[tomaszchacuk]

@jkotas

Unhandled Exception:
System.AggregateException: One or more errors occurred. ---> System.DllNotFoundE xception: WS2_32.dll
at (wrapper managed-to-native) Microsoft.AspNetCore.Server.Kestrel.Internal.Ne tworking.Libuv+NativeMethods:WSAIoctl (intptr,int,int_,uint,int_,int,uint&,intpt r,intptr)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_bind_wind ows_extras (Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle handle) [0x0001f] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_bind (Mic rosoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle handle, Microso ft.AspNetCore.Server.Kestrel.Internal.Networking.SockAddr& addr, System.Int32 fl ags) [0x00023] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.Bind (M icrosoft.AspNetCore.Server.Kestrel.ServerAddress address) [0x0004a] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListenerPrimary.Create ListenSocket () [0x00040] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener+<>c. b__6_0 (System.Object state) [0x00012] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 1,2,3,4,5
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] i n :0
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Thre ading.Tasks.Task task) [0x0004e] in :0
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNoti fication (System.Threading.Tasks.Task task) [0x0002e] in :0
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.T asks.Task task) [0x0000b] in :0
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwait er.GetResult () [0x00000] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.ListenerPrimary+d__11.MoveNext () [0x000f5] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 5
[MVID] ef160044a2e841e9b5a8b2ccca1a98ad 0,1,2,3,4
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskC anceledExceptions) [0x00014] in :0
at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System. Threading.CancellationToken cancellationToken) [0x00052] in :0
at System.Threading.Tasks.Task.Wait () [0x00000] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.CreateServer (Mi crosoft.AspNetCore.Server.Kestrel.ServerAddress address) [0x000e1] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 3
[MVID] ef160044a2e841e9b5a8b2ccca1a98ad 0,1,2
---> (Inner Exception #0) System.DllNotFoundException: WS2_32.dll
at (wrapper managed-to-native) Microsoft.AspNetCore.Server.Kestrel.Internal.Ne tworking.Libuv+NativeMethods:WSAIoctl (intptr,int,int_,uint,int_,int,uint&,intpt r,intptr)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_bind_wind ows_extras (Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle handle) [0x0001f] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_bind (Mic rosoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle handle, Microso ft.AspNetCore.Server.Kestrel.Internal.Networking.SockAddr& addr, System.Int32 fl ags) [0x00023] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.Bind (M icrosoft.AspNetCore.Server.Kestrel.ServerAddress address) [0x0004a] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListenerPrimary.Create ListenSocket () [0x00040] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener+<>c. b__6_0 (System.Object state) [0x00012] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 1,2,3,4,5
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] i n :0
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Thre ading.Tasks.Task task) [0x0004e] in :0
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNoti fication (System.Threading.Tasks.Task task) [0x0002e] in :0
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.T asks.Task task) [0x0000b] in :0
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwait er.GetResult () [0x00000] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.ListenerPrimary+d__11.MoveNext () [0x000f5] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 5
[MVID] ef160044a2e841e9b5a8b2ccca1a98ad 0,1,2,3,4<---

[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what we nt wrong

[jkotas]

This looks like you are running with Windows-specific build of System.Runtime.InteropServices.RuntimeInformation.dll. You need to get the Linux-specific build - when you are publishing the standalone app to copy over, make sure to publish it for Linux.

[tomaszchacuk]

In this case you mean publishing on Windows (donet publish ...) for Linux , right ?

[jkotas]

Right.

[tomaszchacuk]

ok after publishing to linux I have another exception:

System.TypeInitializationException: The type initializer for 'Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.Constants' threw an exception. ---> System.DllNotFoundException: System.Native
at (wrapper managed-to-native) Interop+Sys:GetUnixNamePrivate ()
at Interop+Sys.GetUnixName () [0x00000] in :0
at System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform (System.Runtime.InteropServices.OSPlatform osPlatform) [0x00009] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.Constants.GetECONNRESET () [0x00005] in :0
at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.Constants..cctor () [0x00000] in :0
[MVID] 7fa2145c3191420fad86bd0c6af65de7 3,4
[MVID] 862459bd886947438b3fcf70c862f252 1,2
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Hosting.Internal.WebHost.Start () [0x00053] in :0
at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (Microsoft.AspNetCore.Hosting.IWebHost host, System.Threading.CancellationToken token, System.String shutdownMessage) [0x00002] in :0
at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run (Microsoft.AspNetCore.Hosting.IWebHost host) [0x0002e] in :0
at ConsoleApplication.Program.Main (System.String[] args) [0x00016] in :0
[MVID] 0816638d468e4be5bd8e0692994db5f5 3
[MVID] 151cad73211e43ce93bfdfc0a194d607 0,1,2
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong

built by :

dotnet publish -r ubuntu.16.04-x64

In publish directory i have files : System.Native.as and System.Natiuve.so
Any further ideas ?

[jkotas]

System.*.Native libraries are built in corefx repo: https://github.com/dotnet/corefx/tree/master/src/Native.

[jkotas]

Replace the ones build for x64 that you got from publish with ones built for ARM.

[tomaszchacuk]

Ok. Seems obvious. Should i compile the core fx myself for arm ? I cannot build it for arm myself (errors). Are they somewhere in ready built version ?

[jkotas]

I do not think there is good place to download Linux arm build of corefx. You just need to build the native part (./build.sh native). Are you getting errors in that? I was able to build it myself for Linux arm fine some time ago.

[tomaszchacuk]

Ok build succeded but after copying it to the publish folder and running gives the same effect - same stack trace (corefx updated to the latest git version)

[jkotas]

You can try setting LD_DEBUG to see why it is failing to load.

[tomaszchacuk]

I made this working. The solution was as obvious as it could be. Just rename build System.Native.so to libSystem.Native.so. All in all I have working Kestrel Hello World for Linux ARM Mono

[jkotas]

@tomaszchacuk It is great news! Let us know if there are other issues you run into.

The system is supposed to work without renames like this - something we will need to look into at some point.

[tomaszchacuk]

Any ideas or clues how to be able to debug remotly from Raspbbery Pi using mono? :-) As far as I managed to digg through the topic is neraly impossible but maybe I am missing sth.

[manu-st]

Using ssh/lldb might be easier, however I haven't check lately the status of the sos plugin for lldb on arm (I remember it required a specific version of lldb). There is also the possibility to perform a remote session but I'm not sure if this is working (see dotnet/runtime#5713)

[mkborg]

lldb shall be at least 3.8 to support ARM.

[leemgs]

lldb shall be at least 3.8 to support ARM.

@chunseoklee , Please, Take a look at this.

[mkborg]

@leemgs @chunseoklee see issue dotnet/runtime#5598 for details about lldb 3.8 for ARM

[tomaszchacuk]

Needed to rebuild everything again. When building corefx native I get:

Restoring BuildTools version 1.0.26-prerelease-00704-01...
/home/nexthome/corefx/init-tools.sh: line 78: /home/nexthome/corefx/Tools/dotnet cli/dotnet: cannot execute binary file: Exec format error
ERROR: Could not restore build tools correctly. See '/home/nexthome/corefx/init- tools.log' for more details.
Initializing BuildTools...
/home/nexthome/corefx/init-tools.sh: line 84: /home/nexthome/corefx/packages/Mic rosoft.DotNet.BuildTools/1.0.26-prerelease-00704-01/lib/init-tools.sh: No such f ile or directory
ERROR: An error occured when trying to initialize the tools. Please check '/home /nexthome/corefx/init-tools.log' for more details.

I built coreclr successfully . I remember it was enough to build corefx but id doesnt work.

[jkotas]

The build system in corefx was changed. To build native parts, call build-native.sh directly for cross-compilation or on platforms where a working .NET Core + CLI is not available as download.

[tomaszchacuk]

What do you mean by "directly for cross-compliation" ?

[jkotas]

I meant the change done recently by @maririos : https://github.com/dotnet/corefx/commit/bb56ba4f3e1fa45388720922ad3981ab4e235c82

[qmfrederik]

I got ASP.NET Core up & running on a Raspberry Pi following the instructions in this thread - thanks!

The info on how to compile for ARM seems to be a bit scattered, so I thought I'd leave a note here on how to build for ARM, for the next person looking to do this :):

• Install Linux on the Raspberry Pi 3 On the Raspberry Pi 3, the distribution I'm using is Ubuntu Mate 16.04, they have an image for the Raspberry Pi 2 & 3
• Cross-build CoreCLR and CoreFX for ARM I've built CoreCLR and CoreFX (native only) on 64-bit Ubuntu 16.04 using the cross-build instructions for each repository. It turns out that the script which generates the rootfs doesn't support Ubuntu 16.04 (xenial) yet, and that it doesn't reference lldb-3.8 either (which this thread states is required). PR dotnet/coreclr#7085 . Hint: you can share the rootfs for CoreFX and CoreCLR (saves some time), if you do, use the CoreCLR rootfs because it contains additional packages
• Copy my app to the Raspberry Pi 3 I've published my app for Ubuntu 16.04 (x64), and copied it to the Raspberry Pi
  -Copy .NET Core to the Raspberry Pi 3 I've copied the output of the CoreFX and CoreCLR builds to the Raspberry Pi, and then replaced the runtime files (corerun, .so,...) in my app folder with the ones from my build
• Build libuv for ARM Built libuv using the instructions provided by @leemgs and copied out/Release/lib.target/libuv.so.1 as libuv.so to my app folder
• Finally, ./corerun <myapp>dll starts the app

Because I think I'll have to repeat this excercise a couple of times, I've started with a Vagrant built script that takes care of setting up the Ubuntu VM and cross-building .NET Core in that VM. I'll probably work it a bit the coming weeks to enhance it.

As a side-note:

• Is there a way to create NuGet packages for the ARM runtime that I could host in a private feed? I'm guessing that that way, I could just publish for ARM from 64-bit Ubuntu?
• Are there any runtime ids for ARM Linux in .NET Core?
• Is there a specific reason the cross-build uses an ARM-based chroot instead of a multi-arch Ubuntu install? I'd assume multi-arch cross-compile would be (slightly) faster?

[jkotas]

cc @ellismg

[jkotas]

@qmfrederik Thanks a lot for writing these instructions down!

[leemgs]

Because I think I'll have to repeat this excercise a couple of times, I've started with
a Vagrant built script that takes care of setting up the Ubuntu VM and cross-building .NET Core
in that VM. I'll probably work it a bit the coming weeks to enhance it.

Thanks a million. It will be helpful to vagrant users that want to use .NET Core on Ubuntu MATE 16.04 + Raspberry Pi3 (ARM).

[aviviadi]

I had success with building coreclr (+corefx) in Debug and running complex program.
However, building coreclr in 'release' gives failure (seg fault) when trying to run a program.
I read https://github.com/dotnet/coreclr/issues/4583 and https://github.com/dotnet/coreclr/pull/5396/files?diff=unified

1. Do we _have_ to use the above w/a ?
2. Can cross compilation of coreclr be used with higher clang version ?

[lemmaa]

1. Yes.
2. We are already using clang 3.8 for arm cross build with debugging support. Please reference the https://github.com/lemmaa/help/wiki/Building-llvm-3.8.1-for-CoreCLR-development to get the necessary toolchain.

[aviviadi]

@lemmaa Thanks. I'll give 3.8 a chance after I'll have success with building coreclr in release mode (I'll try first with -O1 instead of -O3 and later on I'll give the w/a a try).

[BruceForstall]

With .NET Core 2.1, Linux/ARM is supported. I'm going to close this bug.