Runtime: Build Instructions for Raspberry Pi devices

[kphillisjr]

I was looking over the documentation and noticed that there is no information how to run CoreCLR on the Raspberry Pi.

• [ ] Instructions to build/run CoreCLR on Raspberry Pi 1 series - This SOC features a single ARM1176JZF-S cpu core. This means that the device features the ARM11 Micro-architecture.
• [x] Instructions to build/run CoreCLR on Raspberry Pi 2 series - This SOC features a quad-core Cortex-A7 cpu. This means that these devices feature the ARMv7-A Micro-architecture.
• [x] Instructions to build/run CoreCLR on Raspberry Pi 3 series - This SOC features a quad-core Cortex-A53 cpu. This means that all of these boards feature the ARMv8-A Micro-architecture.
• [x] Create Instructions for above targets using Windows 10 PC.

[janvorli]

@kphillisjr Linux ARM support is still in development

[kphillisjr]

@janvorli, I know that Linux Arm support is a work-in-progress, and I personally would not mind helping with it's development, and I need documentation on how to get CoreCLR running on my arm device to be able to help with the Regression tests. I have the following boards ready to use.

• Raspberry Pi 1 Model B - The exact board is one of the older 256 MB boards.
• Raspberry Pi 2 Model B

[janvorli]

Ah, ok. Just in case you are not aware of that, there is some basic information on cross building for ARM Linux here:
https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md

[kphillisjr]

I was reading over documents and build documentation and noticed something. The Build configuration for ARM appears to be geared towards chips with VFPv3 floating point, this means chips like the ARMv7-A (aka Cortex-A series ) or newer SoC. The ARM1176 series found in the Raspberry Pi Model 1 supports only VFPv2 floating point. As a quick note a good explanation of ARM floating point can be found on debian's wiki.

[RussKeldorph]

@BruceForstall Does the CoreCLR JIT have any support for "A32," i.e. non-Thumb, instruction encoding generation? If not, that would probably be a bigger hurdle to the Pi-1 than lack of VFPv3 support. IIRC, ARM11 is ARMv6 and doesn't support Thumb-2, which is the usual target for the ARM JIT.

[BruceForstall]

The RyuJIT legacy_backend path only supports the Thumb-2 ISA.

[kphillisjr]

For the most part it should be fairly easy to implement a Thumb back-end if the Thumb-2 ISA is already in place. The only major change is that Thumb lacks some of the newer instructions found in Thumb-2. Also, I do know that ARM has a couple of quick reference cards that should help with identifying the instructions that are not available.

• ARM and Thumb-2 Instruction Set Quick Reference Card
• Thumb 16-bit Instruction Set Quick Reference Card
• Vector Floating Point Instruction Set Quick Reference Card

Also as a quick Note, the Raspberry PI 1 supports Thumb, and VFPv2.

[techyian]

Hi,

Just wanted to add to this thread. I've tried building from source on my Raspberry Pi 3 and receive a Seg Fault in corhlpr.h within the struct COR_ILMETHOD referencing line 610. I've followed the build instructions and have also built libunwind from source which includes the required change mentioned to prevent access to inaccessible memory. This is using the latest Raspbian build.

I've also tried building in release mode but this did not succeed either, however it failed in a different location with a different error (will post actual release mode error when I'm back home!). If there's any other info I can provide then please let me know!

Thanks,

Ian

[techyian]

Raised as new issue dotnet/coreclr#5561

[maartenmensink]

I have been trying to get the coreclr build prerequisites installed on raspberry pi 3 following https://github.com/dotnet/coreclr/blob/master/Documentation/building/cross-building.md but i hit a issues when trying to install sudo apt-get install binutils-aarch64-linux-gnu

i get the following error E: Unable to locate package binutils-aarch64-linux-gnu after some googling i am unable to find if the package is even available for the arm architecture.

uname -a Linux raspberrypi 4.4.34-v7+ dotnet/runtime#4216 SMP Wed Nov 23 15:20:41 GMT 2016 armv7l GNU/Linux

[janvorli]

@maartenmensink - you are running 32 bit linux on the Pi3 (see the armv7l, 64 bit would be armv8) . The aarch64 should be used only in the case you were running 64 bit Linux. The fact that the processor is 64 bit doesn't mean the OS needs to be 64 bit and the Raspian and other distros recommended at the raspberry site are all 32 bit ones.

[maartenmensink]

@janvorli sadly the 32bits gives the same result

sudo apt-get install binutils-arm-linux-gnueabihf results in

pi@raspberrypi:~ $ sudo apt-get install binutils-arm-linux-gnueabihf Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package binutils-arm-linux-gnueabihf

so either the default raspbian image doesn't have the proper apt-get sources or the package isnt available for the arm architecture.

[seanshpark]

@maartenmensink , If you are trying to build inside RPi3 (running in 32bit raspbian or ubuntu mate or any other linux) it wouldn't build. It is not ready yet. Please try in x86-64 host (I use Ubuntu 16.04, 64 bit) cross compilation.

[seanshpark]

@maartenmensink , sorry for wrong explanation. I also had same problem with binutils-arm-linux-gnueabihf (I'm using Ubuntu mate 16.04). I think you need to install clang-3.6 or above. I'm trying with cross built clang 3.8 in my RPi3 and waiting the build...

./build.sh skipnuget skiptests

I think you may need to install these.

sudo apt-get install  build-essential libunwind8-dev clang-3.7

[maartenmensink]

@seanshpark After checking out the repo and running ./build.sh skipnuget skiptests i had to install

sudo apt-get cmake clang-3.5 libicu-dev

but there is another roadblock lldb-3.6-dev isnt available.

pi@raspberrypi:~/projects/coreclr $ ./build.sh skipnuget skiptests
Commencing CoreCLR Repo build
Unsupported CPU armv7l detected, build might not succeed!
Setting up directories for build
Checking prerequisites...
Laying out dynamically generated files consumed by the build system
Laying out dynamically generated Event Logging Test files
Generating Event Logging Tests
Laying out dynamically generated Event Logging Implementation of Lttng
Cleaning the temp folder of dynamically generated Event Logging files
Updating /home/pi/projects/coreclr/bin/obj/Linux.arm.Debug/Generated/eventprovider with /home/pi/projects/coreclr/bin/obj/Linux.arm.Debug/Generated/eventprovider_new
Commencing build of CoreCLR component for Linux.arm.Debug in /home/pi/projects/coreclr/bin/obj/Linux.arm.Debug
~/projects/coreclr/bin/obj/Linux.arm.Debug ~/projects/coreclr
Invoking "/home/pi/projects/coreclr/src/pal/tools/gen-buildsys-clang.sh" "/home/pi/projects/coreclr" 3 5 arm Debug    -DCLR_CMAKE_TARGET_OS=Linux -DCLR_CMAKE_PACKAGES_DIR=/home/pi/projects/coreclr/packages -DCLR_CMAKE_PGO_INSTRUMENT=0
Detected Linux ARM
-- Performing Test HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS
-- Performing Test HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS - Success
-- Looking for ucol_setMaxVariable
-- Looking for ucol_setMaxVariable - not found
CMake Error at src/ToolBox/SOS/lldbplugin/CMakeLists.txt:56 (message):
  Cannot find lldb-3.5, lldb-3.6 or lldb-3.8.  Try installing lldb-3.6-dev
  (or the appropriate package for your platform)


-- Configuring incomplete, errors occurred!
See also "/home/pi/projects/coreclr/bin/obj/Linux.arm.Debug/CMakeFiles/CMakeOutput.log".
See also "/home/pi/projects/coreclr/bin/obj/Linux.arm.Debug/CMakeFiles/CMakeError.log".
~/projects/coreclr
Failed to generate CoreCLR component build project!
pi@raspberrypi:~/projects/coreclr $ sudo apt-get install lldb-3.6-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package lldb-3.6-dev
E: Couldn't find any package by regex 'lldb-3.6-dev'

[seanshpark]

@maartenmensink , SOS is for debugging. You can try (1) disable SOS, (2) build your own clang and lldb.
To disable SOS comment three lines like below. I'm not sure disabling SOS will stop asking lldb.

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2c39780..64a0784 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -549,7 +549,7 @@ if(CLR_CMAKE_PLATFORM_UNIX)
 endif(CLR_CMAKE_PLATFORM_UNIX)

 if(CLR_CMAKE_PLATFORM_UNIX)
-    add_subdirectory(src/ToolBox/SOS/lldbplugin)
+    #add_subdirectory(src/ToolBox/SOS/lldbplugin)
     add_subdirectory(src/pal)
     add_subdirectory(src/coreclr/hosts)
     add_subdirectory(src/ildasm/unixcoreclrloader)
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index d13e8f9..53cd3c3 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -96,7 +96,7 @@ function(add_executable_clr)
 endfunction()

 if(CLR_CMAKE_PLATFORM_UNIX)
-  add_subdirectory(ToolBox/SOS/Strike)
+  #add_subdirectory(ToolBox/SOS/Strike)

   # Include the dummy c++ include files
   include_directories("pal/inc/rt/cpp")
diff --git a/src/ToolBox/CMakeLists.txt b/src/ToolBox/CMakeLists.txt
index 4e7f436..a68b8cb 100644
--- a/src/ToolBox/CMakeLists.txt
+++ b/src/ToolBox/CMakeLists.txt
@@ -1,2 +1,2 @@
-add_subdirectory(SOS)
+#add_subdirectory(SOS)
 add_subdirectory(superpmi)

BTW, which Linux are you using?

[maartenmensink]

@seanshpark i am using RASPBIAN JESSIE LITE

[seanshpark]

@maartenmensink , I've got CoreCLR build success in RPi3. To run any program with this, next step would be to build CoreFX. But unfortunately building native ARM CoreFX may not be possible (or this also I may be wrong?) inside RPi3. For this you may need to build from x86-64 cross compilation.

[janvorli]

I am using clang 3.7 / lldb-3.7-dev to build stuff on my RPI3 with Raspbian jessie.

For corefx native components:
src/Native/build-native.sh arm clang3.7

For coreclr, I needed to patch the build files to support clang3.7 / lldb-3.7 like this:

diff --git a/src/ToolBox/SOS/lldbplugin/CMakeLists.txt b/src/ToolBox/SOS/lldbplugin/CMakeLists.txt
index 1424cff..fc0acd7 100644
--- a/src/ToolBox/SOS/lldbplugin/CMakeLists.txt
+++ b/src/ToolBox/SOS/lldbplugin/CMakeLists.txt
@@ -45,8 +45,8 @@ if(NOT ENABLE_LLDBPLUGIN)
 endif()

 # Check for LLDB library
-find_library(LLDB NAMES LLDB lldb lldb-3.8 lldb-3.6 lldb-3.5 PATHS "${WITH_LLDB_LIBS}" PATH_SUFFIXES llvm NO_DEFAULT_PATH)
-find_library(LLDB NAMES LLDB lldb lldb-3.8 lldb-3.6 lldb-3.5 PATH_SUFFIXES llvm)
+find_library(LLDB NAMES LLDB lldb lldb-3.8 lldb-3.7 lldb-3.6 lldb-3.5 PATHS "${WITH_LLDB_LIBS}" PATH_SUFFIXES llvm NO_DEFAULT_PATH)
+find_library(LLDB NAMES LLDB lldb lldb-3.8 lldb-3.7 lldb-3.6 lldb-3.5 PATH_SUFFIXES llvm)
 if(LLDB STREQUAL LLDB-NOTFOUND)
     if(REQUIRE_LLDBPLUGIN)
         message(FATAL_ERROR "Cannot find lldb-3.5, lldb-3.6 or lldb-3.8. Try installing lldb-3.6-dev (or the appropriate package for your platform)")
@@ -65,6 +65,8 @@ find_path(LLDB_H "lldb/API/LLDB.h")
 if(LLDB_H STREQUAL LLDB_H-NOTFOUND)
     find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.8/include")
     if(LLDB_H STREQUAL LLDB_H-NOTFOUND)
+    find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.7/include")
+    if(LLDB_H STREQUAL LLDB_H-NOTFOUND)
         find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.6/include")
         if(LLDB_H STREQUAL LLDB_H-NOTFOUND)
             find_path(LLDB_H "lldb/API/LLDB.h" PATHS "/usr/lib/llvm-3.5/include")
@@ -78,6 +80,7 @@ if(LLDB_H STREQUAL LLDB_H-NOTFOUND)
             endif()
         endif()
     endif()
+    endif()
 endif()

 message(STATUS "LLDB_H: ${LLDB_H}")

And then build:
./build.sh arm skipgenerateversion skipmscorlib clang3.7

[maartenmensink]

I have been able to get the build working with on RPi3 raspbian jessie

I used the following packages

sudo apt-get update
sudo apt-get install libunwind8 libunwind8-dev libicu-dev cmake clang-3.7 lldb-3.7 lldb-3.7-dev uuid-dev 

and the based on the instructions of @janvorli to include clang-3.7 in de CMakeList.txt

I have create a fork with the changes from the instructions so anybody change use it.

git clone https://github.com/SpringIT/coreclr
cd coreclr
./build.sh arm skipgenerateversion skipmscorlib clang3.7

p.s. dont use a 8gb sd last build failed at 98% no more disk space oops fail

[janvorli]

@maartenmensink the qemu qemu-user-static binfmt-support and debootstrap are not needed. These are for the cross build on x64 Linux, I believe.

[roterdam]

How about building for aarch64. Should that work? It seems Microsoft Windows has AArch64 support which means theoretically it should work since the ABI is same.

[janvorli]

aarch64 doesn't work for Linux yet. There is some work done here https://github.com/cydhaselton/coreclr/issues/1 to enable ARM64 on Android from where we could pull out some of the missing ASM helpers and possibly few other changes and take that as a starting point for enabling aarch64 Linux. I can selectively move those changes to the main branch if there is someone interested in working on the bringup.

[maartenmensink]

The build pipeline seems to be working properly on the RPi3.

Now it's just compile errors. source based on commit https://github.com/dotnet/coreclr/commit/fb3270f7756830f8f055b4c76831f684908463cb

[ 21%] Building ASM object src/vm/wks/CMakeFiles/cee_wks.dir/__/arm/asmhelpers.S.o
<instantiation>:2:18: error: invalid reassignment of non-absolute variable '__FakePrologName'
__FakePrologName="DelayLoad_Helper_Obj_FakeProlog"
                 ^
/home/pi/projects/coreclr/src/vm/arm/asmhelpers.S:1446:5: note: while in macro instantiation
    DynamicHelper 1, _Obj
    ^
<instantiation>:2:18: error: invalid reassignment of non-absolute variable '__FakePrologName'
__FakePrologName="DelayLoad_Helper_ObjObj_FakeProlog"
                 ^
/home/pi/projects/coreclr/src/vm/arm/asmhelpers.S:1447:5: note: while in macro instantiation
    DynamicHelper 1 | 2, _ObjObj
    ^
src/vm/wks/CMakeFiles/cee_wks.dir/build.make:4790: recipe for target 'src/vm/wks/CMakeFiles/cee_wks.dir/__/arm/asmhelpers.S.o' failed
make[2]: *** [src/vm/wks/CMakeFiles/cee_wks.dir/__/arm/asmhelpers.S.o] Error 1
make[2]: *** Waiting for unfinished jobs....

[janvorli]

@maartenmensink ah, I am sorry, I've forgotten to mention that I have also removed the line
__FakePrologName="DelayLoad_Helper_Obj_FakeProlog"
It should not be there. If you remove it, you'll get a successful build.

[maartenmensink]

I was able to do a full build of coreclr based on the above input with commit https://github.com/SpringIT/coreclr/commit/5bb296a8484c8d04bcc196fecaf932b88aa096ac

I will create a clean setup script for raspian jessie based on the input and experience above so other people can have a go at it.

i can recommend the option skiptests to improve build speed. Next up corefx

[janvorli]

I would also like to commit both the changes (support for clang3.7 and the __FakePrologName="DelayLoad_Helper_Obj_FakeProlog" thing) to master. Would you be willing to create a PR for that or would you prefer me doing it?

[maartenmensink]

Yeah i am willing to help out!

What are the PR guidelines i have to conform to?

[janvorli]

Just create a PR with those changes, I will review and merge it after the CI lab tests finish. If it is your first commit, it would ask you to electronically sign CLA and that's all.

[maartenmensink]

PR submitted and CLA signed https://github.com/dotnet/coreclr/pull/8904

[jschwartzenberg]

It seems the build script doesn't work well for ARM. When I run it on Arch like this:

./build.sh arm

It still shows:

Unknown CPU armv6l detected, configuring as if for x64

[janvorli]

@jschwartzenberg are you running on armv7 or armv8 device like RPi2 or 3? We don't support anything else due to the different instruction encoding and the armv6l from your message seems to indicate that it is RPi1 or RPi Zero or some other ARM board with armv6.

[jschwartzenberg]

@janvorli you're right, it's a RPi1. I guess it would be nice if the script could detect that case and point out it's not supported (for now?).

[BruceForstall]

@kphillisjr Can this issue be closed? We have good arm32 and arm64 support now.

[kphillisjr]

@BruceForstall This issue should not have been closed yet. The ARM32 support is still incomplete. It's taking me a while to post a comment because I haft to gather the information to provide a decent update.

[kphillisjr]

I just finished getting a quick todo list for finishing this task. In general the VFP engine is backwards compatible with older models so the changes to the JIT code should be trivial at best. The following tasks should cover most of the popular ARM chips and also ensure that certain edge cases work as well.

• [ ] Expand the JIT VFP Code to handle Neon-less ARM Chips ( See following tasks). In general this covers chips like the ARMv7-R chips ( see: Cortex-R4, Cortex-R5, Cortex-R7, and Cortex-R8), and certain Cortex A9 chips like the Nvidia Tegra 2.
• [ ] Add support for the VFPv3-D16 Floating point variant. This is the VFPv3 floating point unit with only 16 registers, and is found on ARMv7-R chips, and Nvidia Tegra 2.
• [ ] Add support for VFPv2 Floating Point Variant. In general this works like the VFPv3-D16 Floating Point Core with the Notes.
  
  
  
  • The VFPv2 chip implements Floating point exception trapping in hardware. The VFPv3 chip does not feature this.
  
  
  • The VFPv2 does not preserve sign when Denormals Occur. Instead the system flushes to positive zero.
  
  
  • The VFPv2 chip lacks Fixed Point to Floating Point conversion instructions.
  
  
  • The VFPv2 does not support constants in the VMOV instruction.
  
  
  • The VFPv2 core cannot handle 8-bit and 16-bit scalar conversions, but 32-bit conversions work fine.
  
  
• [ ] add support for the VFPv4 and VFPv4-D16 variants. These are the same as the VFPv3 models, but additional features as follows.
  
  
  
  • Fused Multiply Add Instructions.
  
  
  • Half-Precision Support as an Instruction format. Originally this was an optional feature to the VFPv3 engine.
  
  

Note: The first generation Raspberry Pi should be fully functional once the VFPv2 (aka VFP11) changes are implemented. Most of the fixes involve ensuring that the newer Features of the VFPv3+ are not generated when trying to run on a system built for VFPv2.

[BruceForstall]

This is a useful list to have, although perhaps it deserves its own "feature" issue, and not be part of a "Build Instructions" issue, which one would think covers simply how to build and run the source code.

The VFPv3-D16 question came up with https://github.com/dotnet/coreclr/issues/17043, also.

I haven't looked, but don't think the JIT currently generates NEON instructions. Given all the recent activity improving SIMD and hardware intrinsics support for other platform, one can imagine also adding that support, and NEON instruction generation, for ARM, if it is desired.

Overall, it seems likely that the implementation of any backward-looking support (that is, for processors more limited, or older, than what is currently supported) might need to come from the community. Improving CoreCLR support for newer processors seems more likely to be funded by Microsoft, or at least considered among other priorities.

[franksinankaya]

Support for the VFPvX-D16 Floating point variant is now available on coreclr in case you missed it.

See https://github.com/dotnet/runtime/blob/master/docs/workflow/building/coreclr/cross-building.md#cross-compiling-coreclr-for-other-vfp-configurations

Cross compiling CoreCLR for Other VFP configurations

[jkotas]

ARM v6 support is tracked by https://github.com/dotnet/runtime/issues/7764.

Raspberry 2 and 3 work fine.

If there is anything remaining worth spending time on that is not addressed by https://github.com/dotnet/runtime/blob/master/docs/workflow/building/coreclr/cross-building.md#cross-compiling-coreclr-for-other-vfp-configurations, please open a new specific issue on it.