Runtime: Enable RyuJIT Backend for ARM32

[YongseopKim]

We( @YongseopKim; @chunseoklee ; @hseok-oh ; @hqueue ) are going to work for enabling RyuJIT backend for arm32.

Any information that we should know? (like that somebodies are working some feature of RyuJIT on arm32)

[jkotas]

cc @dotnet/jit-contrib

[RussKeldorph]

Awesome! It should be much easier to do it now that x86 paved the 32-bit path. I'm not aware of anyone else working on RyuJIT/ARM32. dotnet/jit-contrib have certainly talked about it a great deal, so expect feedback and feel free to ask questions.

[mskvortsov]

@hqueue I'm working on CodeGen::genSetRegToCond right now.
And I've put my head into the !isRegPairType(tree->TypeGet()) issue already.

(BTW, broadly speaking, there are still quite a few overlapping areas in the bring up process, so at this stage we probably should not try too hard not to repeat each other's efforts).

[hqueue]

@mskvortsov Cool. For!isRegPairType(tree->TypeGet()) and assignedInterval != nullptr , I think they are all related to types which requires two registers , e.g. long and double type in all phases including LSRA.
BTW I will look into other assertions.

[hqueue]

The description is deprecated by Jan. 18, 2017. You can find test result from dotnet/coreclr#8724.

---

Summarize current status with PR dotnet/coreclr#8710 and dotnet/coreclr#8724 (cited from dotnet/coreclr#8724)

Status

Test result from 139 tests from JIT/CodeGenBringUpTests with COMPlus_JitDisasm=BringUpTest:* COMPlus_AltJit=BringUpTest:*

with PR dotnet/coreclr#8710

=======================
Test Results
=======================
# CoreCLR Bin Dir :
# Tests Discovered : 139
# Passed : 61
# Failed : 78
# Skipped : 0
=======================

All 78 failures are Assert failure as follows.

• 54 of 'NYI: float argument' (gone)
• 7 of 'NYI: genSetRegToCond'
• 6 of 'NYI: Lowering of long register argument'
• 4 of 'NYI_ARM: TreeNodeInfoInit default case'
• 2 of 'NYI_ARM: ARM IsContainableImmed for floating point type' (gone)
• 2 of 'NYI_ARM: fgMorphMultiregStructArgs'
• 1 of 'NYI: Cast'
• 1 of 'NYI: ARM genCallFinally'
• 1 of '!isRegPairType(tree->TypeGet())'

with PR dotnet/coreclr#8710 and this PR dotnet/coreclr#8724 (15 more tests are PASSED.)

=======================
Test Results
=======================
# CoreCLR Bin Dir : 
# Tests Discovered : 139
# Passed : 76
# Failed : 63
# Skipped : 0
=======================

All 63 failures are Assert failure as follows.

• 29 of 'assignedInterval != nullptr' (NEW)
• 7 of 'NYI: GT_DIV' (NEW)
• 2 of '!"Unexpected instruction"' (NEW)
• 2 of 'NYI_ARM: Lowering for cast from float' (NEW)
• 1 of 'NYI_ARM: GT_LEA: index and cns are not nil' (NEW)
• 7 of 'NYI: genSetRegToCond' (no change)
• 6 of 'NYI: Lowering of long register argument' (no change)
• 4 of 'NYI_ARM: TreeNodeInfoInit default case' (no change)
• 2 of 'NYI_ARM: fgMorphMultiregStructArgs' (no change)
• 1 of 'NYI: Cast' (no change)
• 1 of 'NYI: ARM genCallFinally' (no change)
• 1 of '!isRegPairType(tree->TypeGet())' (no change)

[mskvortsov]

@hqueue dotnet/coreclr#8731 (on top of your FP work) makes 6 more tests passed.

[hqueue]

@mskvortsov Cool! Now I'm not working on assertions right now, I think you can continue as you wish :) I will run tests tomorrow again.

[YongseopKim]

CoreCLR: 9a870919addaf32de0e6b89e4bf23a893cb42085 + https://github.com/dotnet/coreclr/pull/8724

Test result of JIT/CodeGenBringUpTests with COMPlus_JitDisasm=BringUpTest:* COMPlus_AltJit=BringUpTest:* is as follow.

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 139
# Passed           : 88
# Failed           : 51
# Skipped          : 0
=======================

[hqueue]

Environment:
  branch: master
  coreclr: 19e1cd299da301d1b2cb18ae8479595dbfc7224c
  corefx : 808c9de28abfc4af11b7f3144aab35c1bf48fb36

Device: RPI3
  Test: --testDir=JIT/CodeGenBringUpTests/
Options:
 COMPlus_AltJit=*
 COMPlus_AltJitAssertOnNYI=0
 COMPlus_JitFuncInfoLogFile=/mnt/albireo/mywork/device/rpi3/unittests/runtest/log.JIT.CodeGenBringUpTests.ryujit.methods

Run test with ryujit and legacyjit as a fallback.

=======================
     Test Results
=======================
# CoreCLR Bin Dir  :
# Tests Discovered : 139
# Passed           : 137
# Failed           : 2
# Skipped          : 0
=======================
8 minutes and 37 seconds taken to run CoreCLR tests.

NYI assertion hit frequencies

hqueue@rpi3:/mnt/albireo/mywork/device/rpi3/unittests/runtest/log.20170315$ grep NYI log.JIT.CodeGenBringUpTests.all.ryujit.methods |sed 's|^.* - NYI ||' |sed 's|^.*src/jit/\(.*\))|\1|' |sort |uniq -c |sort -nr
   6180 lsraarm.cpp:1346 - NYI: lowering struct var
    474 morph.cpp:4659 - NYI_ARM: fgMorphMultiregStructArgs
    382 lsraarm.cpp:1335 - NYI: Unimplemented node type lclHeap
    235 lsraarm.cpp:901 - NYI: lowering double
    187 lsraarm.cpp:900 - NYI: lowering struct
      1 lsraarm.cpp:996 - NYI_ARM: overflow checks
      1 codegenarm.cpp:1882 - NYI: genRegCopy

Number of methods falling back to legacy JIT

hqueue@rpi3:/mnt/albireo/mywork/device/rpi3/unittests/runtest/log.20170315$ grep NYI log.JIT.CodeGenBringUpTests.all.ryujit.methods |sed 's|- NYI.*$||' |sort -u |wc -l
158

Number of methods compiled with the RyuJIT backend

hqueue@rpi3:/mnt/albireo/mywork/device/rpi3/unittests/runtest/log.20170315$ grep -v NYI log.JIT.CodeGenBringUpTests.all.ryujit.methods |sort -u |wc -l
1108

[BruceForstall]

@YongseopKim @chunseoklee @hseok-oh @hqueue @mskvortsov @sjsinju

I have a general question: for RyuJIT/ARM32, how can I tell what is being worked on, and who is working on what?

A simple query of GitHub issues shows just a few RyuJIT/ARM32 issues.

There is a RyuJIT/ARM32 GitHub project, but it is little used: https://github.com/dotnet/coreclr/projects/4

All RyuJIT/ARM32 pull requests seem to tag this "meta-issue", as shown here. But that only shows work that is done or close to done, not work that is just starting, or assigned, for example, by assigning individual issues out to individuals when they start working on them.

Would it be reasonable to have the following process:

1. Create GitHub issues for individual work items. This could be immediately when the issue starts being worked on. However, preferably, a whole set of issues would be created in advance for all the work that is known to be required, and we continuously add to that set as new required work is discovered.
2. When someone starts work on a work item, assign that GitHub issue to themselves.
3. The PR can be marked as Fixes dotnet/runtime#4367 (for example), so GitHub will automatically close the associated issue when the PR is merged.

Comments?

[BruceForstall]

Also, @alpencolt

[hqueue]

@BruceForstall I also agree with your suggestions and this is what I think should be :)

However for the Ryuji/ARM32 project https://github.com/dotnet/coreclr/projects/4, I've already know the existence from Jan.(I think it was made around last December, maybe.), but I'don't have any permission to use it. Do you know who made this ? It would be convenient if we can use the project too, in addition to the link to this meta-issue which is used to track the status of ryujit/arm32 overall.

Basically I think exploiting the issue in github is a right way and we should make use of this. One minior problem is that we can't assign the issue to oneself without other's help.

[sjsinju]

Current status of run as CodeGenBringUpTests

Environment:
  coreclr: 62849aabbad290a81349cc2a642f3bb240677c7f
  corefx : 42029ac4c6bda9df2f3d541e8088c837dc3a7b16

Device: RPI3
Test: --testDir=JIT/CodeGenBringUpTests/
Options:
 COMPlus_AltJit=*
 COMPlus_AltJitAssertOnNYI=0

Run test with ryujit and legacyjit

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : /home/sjsujinkim/works/dotnet/coreclr/bin/Product/Linux.arm.Debug
# Tests Discovered : 148
# Passed           : 148
# Failed           : 0
# Skipped          : 0
=======================
5 minutes and 22 seconds taken to run CoreCLR tests.

NYI assertion hit frequencies

sjsujinkim@sjsujinkim-rpi3:~/works/dotnet/coreclr/tests$ grep NYI /home/sjsujinkim/works/logs/test/log_ryujit_func |sed 's|^.* - NYI ||' |sed 's|^.*src/jit/\(.*\))|\1|' |sort |uniq -c |sort -nr
    424 lsraarm.cpp:1352 - NYI: Unimplemented node type lclHeap
    268 morph.cpp:4610 - NYI_ARM: fgMorphMultiregStructArgs
    192 lsraarm.cpp:900 - NYI: lowering double
    106 lowerarm.cpp:149 - NYI_ARM: Lowering for GT_STORE_OBJ isn't implemented
      1 lsraarm.cpp:1352 - NYI: Unimplemented node type obj
      1 codegenarm.cpp:1235 - NYI: Unimplemented node type storeObj

Number of methods falling back to legacy JIT

sjsujinkim@sjsujinkim-rpi3:~/works/dotnet/coreclr/tests$ grep NYI /home/sjsujinkim/works/logs/test/log_ryujit_func |sed 's|- NYI.*$||' |sort -u |wc -l
91

Number of methods compiled with the RyuJIT backend

sjsujinkim@sjsujinkim-rpi3:~/works/dotnet/coreclr/tests$ grep -v NYI /home/sjsujinkim/works/logs/test/log_ryujit_func |sort -u |wc -l
1210

[BruceForstall]

I created a new GitHub "Team" for ARM32 work: @dotnet/arm32-contrib . If you are working on RyuJIT/ARM32, please go to https://github.com/orgs/dotnet/teams/arm32-contrib and request to be added to this team.

This should make it easier to know who is currently working on this project, or who is interested in this project, so a single tag of @dotnet/arm32-contrib will reach all such people.

[alpencolt]

@BruceForstall wrote:

please go to https://github.com/orgs/dotnet/teams/arm32-contrib and request to be added to this team.

It shows 404 error, is it OK?

[BruceForstall]

@alpencolt Not sure why you got that. Try again sometime?

[alpencolt]

Tried many times, @mskvortsov also don't have access. It looks only Members have access.

[hqueue]

@alpencolt @mskvortsov I think you'd better ask @Dmitri-Botcharnikov and join .NET foundation organization, to see above link. It seems taht @Dmitri-Botcharnikov is already involved :)

/cc: @lemmaa

[Dmitri-Botcharnikov]

@hqueue : it seems that I have no rights to invite them to join the .NET Foundation.

[lemmaa]

@BruceForstall, Would you please add @alpencolt, @mskvortsov, @ruben-ayrapetyan, and @sergign60 to @dotnet/samsung team?
cc: @richlander

[BruceForstall]

NYI status

I ran the ARM32 altjit over the dotnet/coreclr tests and saw the following NYI (ordered by number of occurrences):

Total NYI: 28456
Top 14 NYIs: count, name
16155, c:\gh\coreclr\src\jit\lsraarm.cpp:230 - NYI: lowering double
5990, c:\gh\coreclr\src\jit\morph.cpp:4610 - NYI_ARM: fgMorphMultiregStructArgs
4452, c:\gh\coreclr\src\jit\lsraarm.cpp:686 - NYI: Unimplemented node type *long
443, c:\gh\coreclr\src\jit\codegenarm.cpp:789 - NYI: Long compare
426, c:\gh\coreclr\src\jit\codegenarm.cpp:234 - NYI: Overflow not yet implemented
260, c:\gh\coreclr\src\jit\lowerarmarch.cpp:286 - NYI_ARM: Lowering for cast from small type to float
228, c:\gh\coreclr\src\jit\lsraarm.cpp:686 - NYI: Unimplemented node type obj
167, c:\gh\coreclr\src\jit\lowerarmarch.cpp:179 - NYI_ARM: Lowering for GT_STORE_OBJ isn't implemented
132, c:\gh\coreclr\src\jit\codegenarm.cpp:427 - NYI: Overflow not yet implemented
108, c:\gh\coreclr\src\jit\codegenarm.cpp:1081 - NYI: Unimplemented node type jump
50, c:\gh\coreclr\src\jit\codegenarm.cpp:1081 - NYI: Unimplemented node type storeObj
29, c:\gh\coreclr\src\jit\lsraarm.cpp:686 - NYI: Unimplemented node type ckfinite
12, c:\gh\coreclr\src\jit\codegenarm.cpp:1081 - NYI: Unimplemented node type gt_long
4, c:\gh\coreclr\src\jit\lsraarm.cpp:686 - NYI: Unimplemented node type initVal

[hqueue]

16155, c:\gh\coreclr\src\jit\lsraarm.cpp:230 - NYI: lowering double 

Most of this NYI is now being addressed by https://github.com/dotnet/coreclr/issues/10972.

[BruceForstall]

NYI status

As before, I ran the ARM32 altjit over the dotnet/coreclr pri0 (default set of) tests and saw the following NYI (ordered by number of occurrences):

Total NYI: 4438
Top 5 NYIs: count, name
3824, c:\gh\coreclr\src\jit\lsraarm.cpp:720 - NYI: Unimplemented node type *long
224, c:\gh\coreclr\src\jit\morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
206, c:\gh\coreclr\src\jit\morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
143, c:\gh\coreclr\src\jit\lsraarm.cpp:720 - NYI: Unimplemented node type obj
41, c:\gh\coreclr\src\jit\codegenarmarch.cpp:392 - NYI: Unimplemented node type jump

This was run on x86 checked on Windows, using:

c:\gh\coreclr> tests\runtest.cmd x86 checked testenv c:\gh\coreclr\tests\protononjit_testenv.cmd

And, uncommenting this line in protononjit_testenv.cmd:

set COMPLUS_JitFuncInfoLogFile=%TEMP%\JitFuncInfoLogFile.txt 

And, finally, postprocessing JitFuncInfoLogFile.txt.

Can someone provide a similar table for all the tests run on hardware?

How many of the tests pass on hardware (and how many fail)?

There is a PR https://github.com/dotnet/coreclr/pull/11710 to fix the "Unimplemented node type jump" issue.

I don't see an issue for the "*long" case.

There are a number of issues listed in the "Backlog" section of the RyuJIT/ARM32 project: https://github.com/dotnet/coreclr/projects/4. What additional issues are missing from that list?

@dotnet/arm32-contrib

[BruceForstall]

More generally, I'm trying to get a sense of:

1. How much work is left,
   2 What kind of work is left,
2. How many people are working on RyuJIT/arm32, and
3. Who is working on what.

[hqueue]

Can someone provide a similar table for all the tests run on hardware?
How many of the tests pass on hardware (and how many fail)?

@BruceForstall Thank you for the sharing. For now, it took too long in real arm32 hardware to run all tests. Even JIT/Methodical alone took more than 2hrs. And we failed to run all tests altogether within 48hrs due to tests which hangs (not analyzed yet.)

I've got only a partial result for JIT/Methodical and JIT/CodeGenBringUpTests as below.

With COMPlus_AltJit=* COMPLUS_AltJitAssertOnNYI=0 options on rpi3 (built in Debug).
The result is very similar to you result from all tests.

JIT/Methodical

coreclr: 9a6125d0071
corefx : a6bfe2f27a5
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 2085
# Passed           : 1868
# Failed           : 112
# Skipped          : 105
=======================
122 minutes and 46 seconds taken to run CoreCLR tests.
NYI assertion hit frequencies
   4597 morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
    403 codegenarmarch.cpp:392 - NYI: Unimplemented node type jump
    130 morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
     97 lsraarm.cpp:720 - NYI: Unimplemented node type obj
      4 lsraarm.cpp:720 - NYI: Unimplemented node type *long
Number of methods falling back to legacy JIT
345
Number of methods compiled with the RyuJIT backend
9823

JIT/CodeGenBringUpTests

coreclr: 9a6125d0071
corefx : a6bfe2f27a5
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 158
# Passed           : 158
# Failed           : 0
# Skipped          : 0
=======================
4 minutes and 40 seconds taken to run CoreCLR tests.
NYI assertion hit frequencies
    131 morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
Number of methods falling back to legacy JIT
4
Number of methods compiled with the RyuJIT backend
1323

I don't see an issue for the "*long" case.

@mskvortsov Does https://github.com/dotnet/coreclr/pull/11526 address "*long" NYI too ?

[hqueue]

1. How much work is left,
   2 What kind of work is left,

1. Who is working on what.

@BruceForstall
For NYI assertions, I think all NYI assertions are now being worked by contributors.

IIRC @hseok-oh is working on following three assertions.

224, c:\gh\coreclr\src\jit\morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
206, c:\gh\coreclr\src\jit\morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
`143, c:\gh\coreclr\src\jit\lsraarm.cpp:720 - NYI: Unimplemented node type obj`.

And already https://github.com/dotnet/coreclr/pull/11541 addressed 143, c:\gh\coreclr\src\jit\lsraarm.cpp:720 - NYI: Unimplemented node type obj.

For 3824, c:\gh\coreclr\src\jit\lsraarm.cpp:720 - NYI: Unimplemented node type *long, I think @mskvortsov addressed at https://github.com/dotnet/coreclr/pull/11526.

Other ther than NYI assertions, there are 112 runtime failure while running JIT/Methodical on rpi3 as shown above. (most of them are due to non-NYI assertions)
I'm now looking into some of them now.

1. How many people are working on RyuJIT/arm32, and

By now @sjsinju, @mskvortsov, @hseok-oh @alpencolt and @hqueue (me) are working on these ryujit/arm32 issues.

[wateret]

@hqueue @BruceForstall
Count me in, I'm going to work on RyuJIT/armel soon.

[hqueue]

https://github.com/dotnet/coreclr/projects/4. What additional issues are missing from that list?

@BruceForstall I missed @wateret who is going to look into armel (ARM_SOFTFP) implementaion for ryujit/arm32. When @wateret file an issue related to ARM_SOFTFP and it should be added to the project.

@wateret Sorry, I counted those who are working on the already known issues.

[BruceForstall]

@hqueue Thanks, this is exactly what I was looking for.

For now, it took too long in real arm32 hardware to run all tests.

It looks like you ran with a Debug build. Have you tried Checked?

Is it just that RPi3 is slow? Is there another platform that would be faster, at least for development?

[mskvortsov]

A checked run usually takes around an hour and a half on Odroid XU4 board, e.g. here are results for the full testsuite run on current master:

=======================
     Test Results
=======================
# CoreCLR Bin Dir  :
# Tests Discovered : 7091
# Passed           : 6014
# Failed           : 354
# Skipped          : 723
=======================
87 minutes and 37 seconds taken to run CoreCLR tests

However, I had to exclude the following tests as they take too much time (several hours in total IIRC):

JIT/Performance/CodeQuality/Burgers/Burgers/Burgers.sh
GC/Scenarios/muldimjagary/muldimjagary/muldimjagary.sh
JIT/jit64/opt/rngchk/RngchkStress3/RngchkStress3.sh

[hqueue]

It looks like you ran with a Debug build. Have you tried Checked?

@BruceForstall You are right. I've not tried Checked yet. It will be much faster for Checked.
I will let you know the result in Checked when ready.

Is it just that RPi3 is slow? Is there another platform that would be faster, at least for development?

I don't think rpi3 is too slow, because we can run all coreclr unittest within 2hrs on rpi3 rpi3.
I think slow result in my environment is due to (1) incompleteness in ryujit/arm32 (maybe hang? or infinite loop? or incorrect gcinfo?) and (2) Debug build.

[hqueue]

@mskvortsov Thanks. I think I should move to Checked build too and exclude those tests.

[BruceForstall]

@mskvortsov Thanks for opening the issues for JIT asserts/crashes (I often use this query). Are these all the asserts/crashes you see in a test run, or just the most common ones?

[mskvortsov]

@BruceForstall The segfaults are all filed. Regarding assertion failures, currently there are two big groups of them: related to struct passing and related to floating-point RA. There are some other non-common failures here or there. Here is the full list sorted by number of occurrences:

     82 Assertion failed 'unreached' emitarm.cpp:3287
     56 Assertion failed 'foundStructArg' morph.cpp:4727
     54 Assertion failed 'varTypeIsSIMD(arg)' lower.cpp:971
     40 Assertion failed '!varTypeIsStruct(call) || call->HasMultiRegRetVal() || callHasRetBuffArg' morph.cpp:5459
     28 Assertion failed 'spillTempSize <= lvaGetMaxSpillTempSize()' lclvars.cpp:6408
     21 Assertion failed '(allRegs(theInterval->registerType) & mask) != 0' lsra.cpp:892
     18 Assertion failed 'genIsValidDoubleReg(regRec->regNum)' lsra.cpp:6334
     14 Assertion failed 'unreached' emitarm.cpp:3048
      8 Assertion failed 'initVal->OperGet() == GT_CNS_INT' morph.cpp:9153
      6 Assertion failed 'offsetFitsInVectorMem(imm)' emitarm.cpp:2702
      6 Assertion failed 'farthestRefPhysRegRecord != nullptr' lsra.cpp:5972
      3 Assertion failed '!assignedInterval->isActive || isSecondHalfReg(physRegRecord, assignedInterval)' lsra.cpp:6861
      1 Assertion failed 'reg1 != reg2' emitarm.cpp:2005
      1 Assertion failed 'isDoubleReg(reg1)' emitarm.cpp:2083
      1 Assertion failed 'info->numSlots == 1' lsraarmarch.cpp:681
      1 Assertion failed 'emitTypeSizes[TypeGet(type)] > 0' emit.h:2365
      1 Assertion failed 'assignedInterval == nextRegRec->assignedInterval' lsra.cpp:6340

[BruceForstall]

On Windows ARM32 (hardware), with RyuJIT/arm32, when I run crossgen System.Private.Corelib.dll I see the following asserts/NYI:

Total asserts: 833
Top 8 asserts: count, name
570, c:\gh\coreclr\src\jit\morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
199, c:\gh\coreclr\src\jit\morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
32, c:\gh\coreclr\src\jit\codegenarmarch.cpp:760 - remainingSize == TARGET_POINTER_SIZE
18, c:\gh\coreclr\src\jit\lower.cpp:971 - varTypeIsSIMD(arg)
8, c:\gh\coreclr\src\jit\emitarm.cpp:2702 - offsetFitsInVectorMem(imm)
4, c:\gh\coreclr\src\jit\morph.cpp:4727 - foundStructArg
1, c:\gh\coreclr\src\jit\emitarm.cpp:3287 - unreached
1, c:\gh\coreclr\src\jit\lclvars.cpp:6408 - spillTempSize <= lvaGetMaxSpillTempSize()

[BruceForstall]

fyi, I use the following variables to generate this:

set COMPlus_AltJit=*
set COMPlus_AltJitNgen=*
set COMPlus_AltJitName=protojit.dll
set COMPlus_NoGuiOnAssert=1
set COMPlus_AltJitAssertOnNYI=0
set COMPlus_AltJitSkipOnAssert=1
set COMPlus_ContinueOnAssert=1
set ComPlus_JitFuncInfoLogFile=c:\temp\ArmFuncInfo.txt

and then process the ArmFuncInfo.txt.

[BruceForstall]

On Windows ARM32 (hardware), with RyuJIT/arm32, I ran all the tests that Windows/arm64 passes on, and saw 10779 pass, 413 fail. I used these environment variables:

set COMPlus_AltJit=*
set COMPlus_AltJitNgen=*
set COMPlus_AltJitName=protojit.dll
set COMPlus_NoGuiOnAssert=1
set COMPlus_AltJitAssertOnNYI=0
set COMPlus_AltJitSkipOnAssert=1
set COMPlus_ContinueOnAssert=1
set ComPlus_JitFuncInfoLogFile=c:\temp\ArmFuncInfo.txt

And saw these NYI/assert:

Top 4 NYIs: count, name
2057, c:\gh\coreclr\src\jit\morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
1558, c:\gh\coreclr\src\jit\morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
159, c:\gh\coreclr\src\jit\codegenarmarch.cpp:393 - NYI: Unimplemented node type obj
79, c:\gh\coreclr\src\jit\codegenarm.cpp:297 - NYI_ARM: struct return

Top 28 asserts: count, name
2720, c:\gh\coreclr\src\jit\emitarm.cpp:3287 - unreached
643, c:\gh\coreclr\src\jit\morph.cpp:15829 - (call->IsTailCallViaHelper() && (compCurBB->bbJumpKind == BBJ_THROW)) || (call->IsFastTailCall() && (compCurBB->bbJumpKind == BBJ_RETURN) && (compCurBB->bbFlags & BBF_HAS_JMP))
415, c:\gh\coreclr\src\jit\lsra.cpp:892 - (allRegs(theInterval->registerType) & mask) != 0
262, c:\gh\coreclr\src\jit\morph.cpp:4727 - foundStructArg
73, c:\gh\coreclr\src\jit\codegenarmarch.cpp:760 - remainingSize == TARGET_POINTER_SIZE
73, c:\gh\coreclr\src\jit\lsra.cpp:5972 - farthestRefPhysRegRecord != nullptr
53, c:\gh\coreclr\src\jit\lclvars.cpp:6408 - spillTempSize <= lvaGetMaxSpillTempSize()
35, c:\gh\coreclr\src\jit\lower.cpp:971 - varTypeIsSIMD(arg)
32, c:\gh\coreclr\src\jit\codegenarmarch.cpp:691 - gcPtrCount == 0
31, c:\gh\coreclr\src\jit\morph.cpp:5459 - !varTypeIsStruct(call) || call->HasMultiRegRetVal() || callHasRetBuffArg
20, c:\gh\coreclr\src\jit\emitarm.cpp:2702 - offsetFitsInVectorMem(imm)
13, c:\gh\coreclr\src\jit\lir.cpp:1469 - node->IsLIR()
12, c:\gh\coreclr\src\jit\emitarm.cpp:3048 - unreached
3, c:\gh\coreclr\src\jit\lsra.cpp:6334 - genIsValidDoubleReg(regRec->regNum)
3, c:\gh\coreclr\src\jit\lsra.cpp:6861 - !assignedInterval->isActive || isSecondHalfReg(physRegRecord, assignedInterval)
1, c:\gh\coreclr\src\jit\emitarm.cpp:203 - isGeneralRegister(id->idReg2())
1, c:\gh\coreclr\src\jit\codegenlegacy.cpp:11105 - !(op1->gtFlags & GTF_REG_VAL)
1, c:\gh\coreclr\src\jit\lsraarmarch.cpp:686 - info->numSlots == 1
1, c:\gh\coreclr\src\jit\codegenlinear.cpp:1158 - tree->gtHasReg()
1, c:\gh\coreclr\src\jit\emitarm.cpp:2083 - isDoubleReg(reg1)
1, c:\gh\coreclr\src\jit\emitarm.cpp:2005 - reg1 != reg2

[BruceForstall]

@dotnet/arm32-contrib I opened GitHub issues for all the NYI and asserts that didn't already have issues. You can see these in the RyuJIT/arm32 project: https://github.com/dotnet/coreclr/projects/4.

You can also see them in a query for arm32 CodeGen issues: https://github.com/dotnet/coreclr/issues?utf8=%E2%9C%93&q=is%3Aopen%20is%3Aissue%20label%3Aarch-arm32%20label%3Aarea-CodeGen.

If you are going to work on one, ideally you could assign it to yourself. Although I think there's a permission problem with assigning issues to non-Microsoft people? In any case, please note in the issue that you are working on it.

[BruceForstall]

If you are going to work on one, ideally you could assign it to yourself. Although I think there's a permission problem with assigning issues to non-Microsoft people? In any case, please note in the issue that you are working on it.

You can also just ask a Microsoft person to assign you the issue, by either tagging me (@BruceForstall) or @RussKeldorph -- we should be able to make the assignment.

[hqueue]

@dotnet/arm32-contrib I opened GitHub issues for all the NYI and asserts that didn't already have issues. You can see these in the RyuJIT/arm32 project: https://github.com/dotnet/coreclr/projects/4.

You can also see them in a query for arm32 CodeGen issues: https://github.com/dotnet/coreclr/issues?utf8=%E2%9C%93&q=is%3Aopen%20is%3Aissue%20label%3Aarch-arm32%20label%3Aarea-CodeGen.

@BruceForstall Thank you for listing up tasks. I could see a tons of issues now.

[hqueue]

We can now run simple console "Hello, world!" with Ryujit only with the latest master branch.

Thank you for all who worked on this issue :)

[CarolEidt]

Congratulations!

[hqueue]

I've ran tests(pri0) with the latest commit on June 5, 2017.

Environment

• On Rpi3
• Checked build
• All tests (pri0)
• With COMPlus_AltJit=* COMPlus_AltJitAssertOnNYI=0 options
• Exclude more than 10 tests which took long time.

coreclrtests.xml.txt (Please rename this file to xml)

coreclr: 170c8257ea6
corefx : 029872d41ee
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 7103
# Passed           : 6066
# Failed           : 294
# Skipped          : 743
=======================
221 minutes and 6 seconds taken to run CoreCLR tests.
NYI assertion hit frequencies
  18960 emitarm.cpp:7666 - NYI: vldr/vstr encoding is not available.
  12987 morph.cpp:4138 - NYI_ARM: Struct split between integer registers and stack
   1378 morph.cpp:3810 - NYI_ARM: Struct can be split between registers and stack
    901 emitarm.cpp:7578 - NYI: vldr/vstr encoding is not available.
    216 codegenarmarch.cpp:393 - NYI: Unimplemented node type obj

Number of methods falling back to legacy JIT
3973
Number of methods compiled with the RyuJIT backend
35893

Failure analysis

• FAILED: 294

  • Non-NYI assertion : 289

  132                    File: /opt/code/github/hqueue/coreclr/src/jit/morph.cpp Line: 4727 Assertion failed 'foundStructArg'
   60                    File: /opt/code/github/hqueue/coreclr/src/jit/morph.cpp Line: 5459 Assertion failed '!varTypeIsStruct(call) || call->HasMultiRegRetVal() || callHasRetBuffArg' in 'HFATest.TestCase:Main():int' (IL size 646)
   52                    File: /opt/code/github/hqueue/coreclr/src/jit/lower.cpp Line: 991 Assertion failed 'varTypeIsSIMD(arg)'
   14                    File: /opt/code/github/hqueue/coreclr/src/jit/lsra.cpp Line: 892 Assertion failed '(allRegs(theInterval->registerType) & mask) != 0'
    8                    File: /opt/code/github/hqueue/coreclr/src/jit/morph.cpp Line: 9184 Assertion failed 'initVal->OperGet() == GT_CNS_INT'
    7                    File: /opt/code/github/hqueue/coreclr/src/jit/lsra.cpp Line: 5972 Assertion failed 'farthestRefPhysRegRecord != nullptr'
    6                    File: /opt/code/github/hqueue/coreclr/src/jit/lsra.cpp Line: 6926 Assertion failed '!assignedInterval->isActive || isSecondHalfReg(physRegRecord, assignedInterval)'
    4                    File: /opt/code/github/hqueue/coreclr/src/jit/lsraarmarch.cpp Line: 702 Assertion failed 'info->numSlots == 1'
    2                    File: /opt/code/github/hqueue/coreclr/src/jit/codegenarmarch.cpp Line: 691 Assertion failed 'gcPtrCount == 0'
    1                    File: /opt/code/github/hqueue/coreclr/src/jit/lsra.cpp Line: 6340 Assertion failed 'assignedInterval == nextRegRec->assignedInterval'
    1                    File: /opt/code/github/hqueue/coreclr/src/jit/emit.h Line: 2370 Assertion failed 'emitTypeSizes[TypeGet(type)] > 0'
    1                    File: /opt/code/github/hqueue/coreclr/src/jit/emitarm.cpp Line: 2451 Assertion failed '!"Instruction cannot be encoded"'
    1                    File: /opt/code/github/hqueue/coreclr/src/jit/codegenlinear.cpp Line: 1158 Assertion failed 'tree->gtHasReg()'
  

  • Segmenataion fault: 1

             ./DevDiv_283795.sh: line 240: 17363 Segmentation fault      (core dumped) $_DebuggerFullPath "$CORE_ROOT/corerun" $ExePath $CLRTestExecutionArguments
  

  • Wrong result: 2

  FAILED   - [4924][   3s]JIT/Methodical/VT/callconv/_il_reljumper4/_il_reljumper4.sh
  FAILED   - [4925][   3s]JIT/Methodical/VT/callconv/_il_reljumper5/_il_reljumper5.sh
  

  • Killed: 2

  FAILED   - [2496][  24s]JIT/jit64/opt/cse/HugeField1/HugeField1.sh
  FAILED   - [2497][  23s]JIT/jit64/opt/cse/HugeField2/HugeField2.sh
  
  

[hqueue]

With recent master branch 5041fa732b48728db01e9875358d6e564bd31548,
we can now run complex HelloWorld.exe with RyuJIT/ARM32 only, i.e. without legacy backend.

And all JIT/CodeGenBringUpTests tests passed with RyuJIT/ARM32 only too.

coreclr: 5041fa732b4
corefx : 4437f0e0719
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 158
# Passed           : 158
# Failed           : 0
# Skipped          : 0
=======================
2 minutes and 33 seconds taken to run CoreCLR tests.

[hqueue]

A month is passed from the last update.

Here is a recent status of pri2 unit test.

Target: Raspberry Pi 2
coreclr : 6ac0222061a (Release build)
corefx : 8affaae021f (Release build)
coreclr tests: built for x86 (Release build)

coreclr: 6ac0222061a
corefx : 8affaae021f
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 11380
# Passed           : 10548
# Failed           : 19
# Skipped          : 813
=======================
328 minutes and 33 seconds taken to run CoreCLR tests.

Above result exclude following tests which requires long execution time in addition to default excluded tests for ARM32 (https://github.com/dotnet/coreclr/blob/master/tests/testsUnsupportedOnARM32.txt).

Additional exclusion list:

JIT/Performance/CodeQuality/Burgers/Burgers/Burgers.sh
GC/Scenarios/muldimjagary/muldimjagary/muldimjagary.sh
JIT/jit64/opt/cse/hugeexpr1/hugeexpr1.sh
JIT/jit64/regress/ddb/118414/118414/118414.sh
JIT/jit64/regress/ddb/113574/113574/113574.sh
JIT/jit64/regress/asurt/143616/foo/foo.sh
JIT/jit64/regress/ddb/103087/103087/103087.sh
JIT/Regression/VS-ia64-JIT/V1.2-M02/b28158/b28158/b28158.sh
JIT/jit64/opt/rngchk/RngchkStress3/RngchkStress3.sh
JIT/jit64/opt/rngchk/SimpleArray_01_o/SimpleArray_01_o.sh
JIT/Performance/CodeQuality/SIMD/ConsoleMandel/ConsoleMandel/ConsoleMandel.sh
JIT/Performance/CodeQuality/Linq/Linq/Linq.sh

[BruceForstall]

Do you have a characterization of the 19 failures? If you run those 19 with a Checked JIT, do they assert? Have you tried to run Release Coreclr but Checked JIT, to enable asserts?

The current status of the outstanding issues should summarized here: https://github.com/dotnet/coreclr/projects/4. There are 10 issues in the "In Progress" column (ignoring this issue), with assigned owners. Are they all being actively worked on?

What is next (besides finishing fixing the existing known issues)?

Have you tried running with JIT stress modes (e.g., COMPlus_JitStress=1, COMPlus_JitStress=2, COMPlus_JitStressRegs, COMPlus_TailcallStress, COMPlus_JITMinOpts=1)? What about GC stress (COMPlus_GCStress=3/4/8/C)?

What about looking at generated code quality (CQ) and throughput (JIT speed -- TP), comparing against LEGACY_BACKEND arm32?

[hqueue]

Do you have a characterization of the 19 failures? If you run those 19 with a Checked JIT, do they assert?

The result is from Release and about 10 tests are known regression (#13225) and remaining failures are due to wrong result. Issues are not created yet.

Have you tried to run Release Coreclr but Checked JIT, to enable asserts?

There are two assertions observed, but this is the first result after fixing, we didn't analyze it yet.

The current status of the outstanding issues should summarized here: https://github.com/dotnet/coreclr/projects/4. There are 10 issues in the "In Progress" column (ignoring this issue), with assigned owners. Are they all being actively worked on ?

AFAIK they are being worked on, and I found that some of them is already resolved. We will check the status.

What is next (besides finishing fixing the existing known issues)?

We are going to run CoreFX tests before running stress test for Jit and GC.
What do you think of it ? Is it worthwhile do CoreFX tests before stress test ?

What about looking at generated code quality (CQ) and throughput (JIT speed -- TP), comparing against LEGACY_BACKEND arm32?

For code quality, there is a issues addressing CQ of RyuJIT/ARM32 (#12971) and minor issues(https://github.com/dotnet/coreclr/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20arm32label%3Aenhancement%20).
For throughput, I didn't evaluated yet.

[BruceForstall]

dotnet/coreclr#13225 has just been fixed, so that should leave just 9 failures. That's great!

We are going to run CoreFX tests before running stress test for Jit and GC. What do you think of it ? Is it worthwhile do CoreFX tests before stress test ?

Seems like a good idea. You could, of course, do both in parallel if you have enough time/machines/people.

[hqueue]

Seems like a good idea. You could, of course, do both in parallel if you have enough time/machines/people.

Sure, why not :) We will try stress tests in parallel as soon as possible.

[hqueue]

With 8e2d570c730419d2f8610a8d01b0a3e0da6cb29f, arm Release build passed all Pri2 tests, although there are assertions observed in Checked build. And remaining assertions will be addressed by opened issues(#13453, dotnet/coreclr#13565) and PR(#13579).

Tested on Raspberry Pi2.
CoreCLR (Release): 8e2d570c730419d2f8610a8d01b0a3e0da6cb29f
CoreFX (Release): 06d361cf2bb
coreclr tests: built for x86 (Release build) (Pri2)

=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 11387
# Passed           : 10574
# Failed           : 0
# Skipped          : 813
=======================

[BruceForstall]

The coreclr CI system now supports testing RyuJIT/arm32 on Windows, as well as arm32 LEGACY_BACKEND on Windows. In the CI, RyuJIT/arm32 is now named "arm", and LEGACY_BACKEND arm32 JIT is named "armlb". There is one more job triggered with every change (for people on the ARM64 "white list"): "Windows_NT arm Cross Checked Build and Test".

With https://github.com/dotnet/coreclr/pull/14055, we can also trigger a number of JitStress/JitStressRegs/TailcallStress/MinOpts test runs for arm32 as well. See https://github.com/dotnet/coreclr/pull/14072 for an example. They haven't all been run yet, so they are probably not error-free. The tests\arm\Tests.lst file should be updated over time to ensure all the runs are clean, and failing tests marked with EXPECTED_FAIL and a tracking GitHub issue number. (These tests can only be triggered by the ARM64 "white list" of approved Microsoft developers.)

[BruceForstall]

The coreclr CI system now supports testing RyuJIT/arm32 on Windows via the altjit mechanism (compiling for ARM on x86 using protononjit.dll). Because it is cross-compilation, the generated ARM code doesn't run; it is thrown away (but the x86 generated code does run). This kind of testing can be useful for doing lots of stress mode compilations, looking for asserts. (Note that I added equivalent test modes for x64-hosted, arm64-targeting testing). One advantage is these modes can be triggered by anyone, not just those on the ARM64 hardware "white list", since they use standard x86/x64 VMs. Also, we have lots of x86/x64 hardware (or VMs), compared to the currently limited number and slow speed of ARM hardware, so these are faster, more reliable, and don't spend so much time waiting, so they shouldn't time out, for instance.

You can see an example of these runs here: https://github.com/dotnet/coreclr/pull/14182.

Note, however, that the runs currently are not clean: there are some failures in every run. I've started adding such asserts and issues to GitHub, summarized in the RyuJIT/arm32 project page: https://github.com/dotnet/coreclr/projects/4.

[BruceForstall]

I've opened GitHub issues corresponding to every assert found in the following Windows RyuJIT/arm32 altjit jobs:

Windows_NT x86_arm_altjit Checked forcerelocs
Windows_NT x86_arm_altjit Checked jitsse2only
Windows_NT x86_arm_altjit Checked jitstress1
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs0x1000
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs0x10
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs0x80
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs1
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs2
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs3
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs4
Windows_NT x86_arm_altjit Checked jitstress2_jitstressregs8
Windows_NT x86_arm_altjit Checked jitstress2
Windows_NT x86_arm_altjit Checked jitstressregs0x1000
Windows_NT x86_arm_altjit Checked jitstressregs0x10
Windows_NT x86_arm_altjit Checked jitstressregs0x80
Windows_NT x86_arm_altjit Checked jitstressregs1
Windows_NT x86_arm_altjit Checked jitstressregs2
Windows_NT x86_arm_altjit Checked jitstressregs3
Windows_NT x86_arm_altjit Checked jitstressregs4
Windows_NT x86_arm_altjit Checked jitstressregs8
Windows_NT x86_arm_altjit Checked minopts
Windows_NT x86_arm_altjit Checked tailcallstress
Windows_NT x86_arm_altjit Checked
Windows_NT x86_arm_altjit Debug
Windows_NT x86_arm_altjit Release

The "corefx" altjit jobs don't work yet.

You can find the issues in project page here or by looking at the arm32 CodeGen issues here

[BruceForstall]

The corefx altjit jobs now work. I've opened 7 new issues found running these tests. The jobs are:

Windows_NT x86_arm_altjit Checked corefx_baseline
Windows_NT x86_arm_altjit Checked corefx_jitstress1
Windows_NT x86_arm_altjit Checked corefx_jitstress2
Windows_NT x86_arm_altjit Checked corefx_jitstressregs0x1000
Windows_NT x86_arm_altjit Checked corefx_jitstressregs0x10
Windows_NT x86_arm_altjit Checked corefx_jitstressregs0x80
Windows_NT x86_arm_altjit Checked corefx_jitstressregs1
Windows_NT x86_arm_altjit Checked corefx_jitstressregs2
Windows_NT x86_arm_altjit Checked corefx_jitstressregs3
Windows_NT x86_arm_altjit Checked corefx_jitstressregs4
Windows_NT x86_arm_altjit Checked corefx_jitstressregs8
Windows_NT x86_arm_altjit Checked corefx_minopts

and can be triggered with any dotnet/coreclr PR. None of the jobs pass right now; besides the asserts, there appear to be some runtime failures that only occur when the altjit is invoked. E.g., you can see all the test runs in this PR: https://github.com/dotnet/coreclr/pull/14334 (choose "View Details" near the bottom.)

[hqueue]

@dotnet/arm32-contrib I'm just curious when are we going to enable RyuJIT as a default compiler for ARM32. Alghouth, it seems that there are several failures in ARM32 in stress mode tests and sometimes there is regressions now, I think it's time to discuss about criteria for enabling RyuJIT, for example pass all CI tests enabled above.
All RyuJIT/ARM32 issues are set to milestone 2.1.0, and I'm just curious about a goal all you have in mind. Please share your thought in mind :)

[BruceForstall]

We've recently been talking about this internally. The answer is "soon". The remaining work is: (1) there are a couple remaining asserts, (2) there are some tests that fail in various stress and non-stress modes -- presumably some because of bad codegen, (3) there are some GC stress issues, (4) we haven't measured throughput or investigated generated code quality compared to legacy backend.

Probably only completing (1) and (2) are all we will wait for to make RyuJIT the default. For GC stress, even legacy backend has several issues. For CQ/TP, we will have to work on those before 2.1 ships.

[BruceForstall]

Note: with https://github.com/dotnet/coreclr/pull/14921, the CI will now use RyuJIT/arm32 to crossgen System.Private.CoreLib.dll in "arm" jobs ("armlb" jobs will continue to use legacy backend for everything).

[hqueue]

We've recently been talking about this internally. The answer is "soon".

Glad to here this :)

The remaining work is: (1) there are a couple remaining asserts, (2) there are some tests that fail in various stress and non-stress modes -- presumably some because of bad codegen, (3) there are some GC stress issues,

Thank you again for reporting issues by enabling a lot of test with various stress combination.

(4) we haven't measured throughput or investigated generated code quality compared to legacy backend.

There is an live issue dotnet/coreclr#12971 before to evaluate performance. But we neither have looked into throughput seriously yet.

Probably only completing (1) and (2) are all we will wait for to make RyuJIT the default. For GC stress, even legacy backend has several issues. For CQ/TP, we will have to work on those before 2.1 ships.

Looks good and feasible for near future. And we also think supporting GC without bug is a never-ending task in managed environments with GC, even in mature JVM.

Note: with dotnet/coreclr#14921, the CI will now use RyuJIT/arm32 to crossgen System.Private.CoreLib.dll in "arm" jobs ("armlb" jobs will continue to use legacy backend for everything).

Thanks for the PR!

[alpencolt]

@BruceForstall I've prepared list of remaining issues for RyuJIT. Could you check it and add more if I've missed something. For making RyuJIT as default compiler we are needed fix (1) and (4), am I right?

1. Assertions:

   • [RyuJIT/armel] Assertion failed 'argValue->TypeGet() == TYP_STRUCT' dotnet/coreclr#14955
   • Test failure: JIT_Directed._coverage_oldtests_lclflddiv_cs_d_lclflddiv_cs_d_/_coverage_oldtests_lclflddiv_cs_d_lclflddiv_cs_d_cmd dotnet/coreclr#14583
   • [ARM] Intermittent segfaults in JIT/Methodical/cctor/misc/threads1_cs_r dotnet/coreclr#12421
   • [ARM32] CompareExchange working wrong with clang3.9 dotnet/coreclr#15074

2. Unimplemented features (and features in progress)

   • [RyuJIT/ARM32] Support fast tail call optimization dotnet/coreclr#13828
3. NYI_ARM (23) dotnet/coreclr#13000:

./jit/lowerarmarch.cpp:       NYI_ARM("ARM IsContainableImmed for floating point type");
./jit/lowerarmarch.cpp:       NYI_ARM("initblk loop unrolling is currently not implemented.");
./jit/lowerarmarch.cpp:       NYI_ARM("Lowering for cast from float to small type"); // Not tested yet.
./jit/lsraarm.cpp:            NYI_ARM("LinearScan::TreeNodeInfoInit for GT_INTRINSIC");
./jit/lsraarm.cpp:            NYI_ARM("TreeNodeInfoInit default case");
./jit/codegenarm.cpp:         NYI_ARM("genCodeForInitBlkUnroll");
./jit/codegenarm.cpp:         NYI_ARM("st.lclFld contained operand");
./jit/codegenarm.cpp:         NYI_ARM("st.lclVar contained operand");
./jit/codegenarm.cpp:         NYI_ARM("NOGC_WRITE_BARRIERS");
./jit/lsraarmarch.cpp:        NYI_ARM("NOGC_WRITE_BARRIERS");
./jit/lsraarmarch.cpp:        NYI_ARM("float reg varargs");
./jit/lsraarmarch.cpp:        NYI_ARM("initblk loop unrolling is currently not implemented.");
./jit/codegenarmarch.cpp:     NYI_ARM("genRegCopy from 'int' to 'float'");
./jit/codegenarmarch.cpp:     NYI_ARM("GT_CMPXCHG");
./jit/codegenarmarch.cpp:     NYI_ARM("genIntrinsic for round - not implemented yet");
./jit/codegenarmarch.cpp:     NYI_ARM("genRegCopy from 'int' to 'float'");
./jit/codegenarmarch.cpp:     NYI_ARM("SIMD genStructReturn");
./jit/codegenlegacy.cpp:      NYI_ARM("LOCK instructions");
./jit/lower.cpp:              NYI_ARM("lower: struct argument by fast tail call");
./jit/emitarm.cpp:            NYI_ARM("JitDataOffset static fields");
./jit/emitarm.cpp:            NYI_ARM("Thread-Local-Storage static fields");
./jit/emitarm.cpp:            NYI_ARM("JitDataOffset static fields");
./jit/emitarm.cpp:            NYI_ARM("Thread-Local-Storage static fields");

1. JitStress:
   https://github.com/dotnet/coreclr/issues?q=is%3Aissue+is%3Aopen+label%3AJitStress+label%3Aarch-arm32
2. GCStress:
   https://github.com/dotnet/coreclr/issues?q=is%3Aissue+is%3Aopen+label%3Aarch-arm32+label%3AGCStress
3. Performance:
   
   
   
   • Need to check benchmarks comparing to legacy backend
   
   
4. Others:
   
   
   
   • Blocking issues for profiler APIs on ARM32/64 dotnet/coreclr#14526
   
   

[CarolEidt]

@BruceForstall can you advise what would be the best place for me to focus? I could work on jitstress or NYIs, or ...?

[alpencolt]

@BruceForstall have you had the time to look into https://github.com/dotnet/coreclr/issues/8496#issuecomment-347195200? Is there any other big fields for work? Or it's difficult to estimate at the current moment?

Another thing which I'm interesting is it possible to generate Linux/ARM code like protononjit but on Windows host (off course 32-bit target on 32-bit host)? If it's not implemented yet would it be difficult to implement it?

[CarolEidt]

Another thing which I'm interesting is it possible to generate Linux/ARM code like protononjit but on Windows host (off course 32-bit target on 32-bit host)?

I believe that's what linuxnonjit.dll does.

[BruceForstall]

@alpencolt Sorry for the slow response. Thank you for collecting the list of issues.

I agree that 1 and 4 should be fixed before we switch RyuJIT/arm32 to be the default. I would also add that "non-stress" runs should also be clean: simple Debug/Checked/Release Pri-1 runs. There are currently 3 test failures in Release. However, I'm willing to be a bit flexible, as well, as there are benefits to doing the switch, and LEGACY_BACKEND is not exactly clean (perhaps I should do a full armlb test run and see how it compares to current RyuJIT results).

As for the issues you list in 1, I'm not sure any need to be blocking:

• dotnet/coreclr#14583: this is LEGACY_BACKEND
• dotnet/coreclr#14955: this is armel only, as far as I know. Should it be blocking?
• dotnet/coreclr#12421, dotnet/coreclr#15074 -- I don't think these should be considered blocking. They don't seem to be RyuJIT specific.

So, to summarize what I think needs to be fixed before conversion:

• dotnet/coreclr#15150 -- [RyuJIT/arm32] Assert failure: pImportSection == pModule->GetImportSectionForRVA(rva)
• dotnet/coreclr#15153 -- [RyuJIT/arm32] Release test failures
• dotnet/coreclr#14860 -- [RyuJIT/arm32][JITMinOpts=1] Test runtime failures

I'm a little worried we haven't yet fully tested ReadyToRun / crossgen: https://github.com/dotnet/coreclr/issues/14916

I think we just disable tests failing in dotnet/coreclr#15155 and dotnet/coreclr#15156 if those aren't fixed when we convert.

Comments?

[alpencolt]

@CarolEidt great news, thank you!

@BruceForstall thank you for response. We will update our plans.

dotnet/coreclr#14955: this is armel only, as far as I know. Should it be blocking?

Up to you. Since Samsung use armel this issue is critical for us.

We haven't tested crossgen a lot too but I've measured launch time of Xamarin applications on Tizen platform for Fragile mode. And RyuJIT is more than 10% slower than Legacy. So I don't know about stability but there is regression in performance unfortunately.

[BruceForstall]

Since Samsung use armel this issue is critical for us.

How much testing has Samsung done with RyuJIT and armel?

The CI job https://ci.dot.net/job/dotnet_coreclr/job/master/job/armel_cross_checked_tizen_prtest only runs 22 tests, and it is using LEGACY_BACKEND. When we switch to RyuJIT/arm32 as the default, it will use RyuJIT. Has Samsung tested these (and more) with RyuJIT/arm32?

[alpencolt]

@BruceForstall we do it but not on regular basis. There are about 2 failed tests from more than 10k called for both ABI. We will rerun them and I''ll provide more detailed results.

[alpencolt]

@BruceForstall sorry for delaying.
Test results for RyuJIT ARM/Linux (failed tests):

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 11402
# Passed           : 10588
# Failed           : 8
# Skipped          : 806
=======================

Test results for RyuJIT ARMEL/Linux (failed tests):

=======================
     Test Results
=======================
# CoreCLR Bin Dir  : 
# Tests Discovered : 11402
# Passed           : 10574
# Failed           : 22
# Skipped          : 806
=======================

Checked versions have about 6 failed tests. There are several segmentation fault and I think some of them because wrong configuration, we will check them manually.

[BruceForstall]

@alpencolt Are those test results from a Checked or Release run? It sounds like maybe Release, since you say at the end that Checked has a different set of failures.

Can you open GitHub issues for the failures, once you have verified them?

[alpencolt]

@BruceForstall release sorry that forget to mention.
Armel tests were executed on chroot and now we've started to check them on real phone and many of this fails cannot be reproduced.
So our current infrastructure isn't ideal. I will share results when check carefully.

[BruceForstall]

As of https://github.com/dotnet/coreclr/commit/d135144e76d15054d62b996e218694c19fdab9de, RyuJIT is now the default JIT for ARM32 in .NET Core!

Congratulations and Thank You to everyone who has contributed to this effort!

You can see from this issue that the RyuJIT/arm32 porting effort started almost exactly one year ago. Starting then, building on the working legacy arm32 JIT and leveraging x86/x64/arm64 RyuJIT work, arm32 has gone from nothing to a fully functioning, extensively tested compiler, working on multiple platforms, and even with multiple ABIs (namely, SOFTFP). It is tested in the CI with over 30 JIT stress modes on Windows ARM hardware, some minimal testing on Ubuntu and Tizen, and almost 40 JIT stress modes in x86_arm altjit mode.

We have more work to do to finish this port and prepare for shipping, including fixing more bugs, including getting GCStress clean, and testing and automating more test configurations. The work continues to be tracked here, and in generic queries for "arch-arm32" labelled bugs.

Note that all Ubuntu and Tizen testing in the CI is now using RyuJIT/arm32. For Windows, the "arm" jobs use RyuJIT/arm32 and the "armlb" jobs use the LEGACY_BACKEND JIT, now named legacyjit.dll.

Onwards!

[YongseopKim]

@dotnet/arm32-contrib thanks for your collaborations. I'll close this because now RyuJIT is the default backend-engine. Happy hacking onwards!

[BruceForstall]

fyi, I am working on adding Ubuntu arm32 testing in the dotnet/coreclr CI system, on hardware, with https://github.com/dotnet/coreclr/pull/17018.

[BruceForstall]

As of today, .NET Core 2.1 has been released, with ARM32 officially supported, using the RyuJIT/ARM32 compiler! You can see the announcement here.

Once again, thank you to everyone who contributed to this effort! It's been a great example of cross-company, around-the-world collaboration.

Based on the opening date of this issue, the RyuJIT/ARM32 port took about 18 months from start to finish. I mentioned some numbers above related to testing; here are some more numbers I collected recently for ARM32 testing done in our CI system:

• 2 jobs on every PR (1 Windows, 1 Ubuntu)
• 62 Windows arm on hardware test jobs in CI, 3 running daily, the rest running weekly.
• 61 Ubuntu arm on hardware jobs in CI, 3 running daily, the rest running weekly.
• 40 Windows x86_arm_altjit test jobs in CI, running daily

These include "normal" as well as many JIT stress and GC stress variations, using both the dotnet/coreclr and dotnet/corefx testbeds. In addition, there was testing on the Tizen emulator.

I'm looking forward to seeing what developers do with this work!