Mono: [ARM] AOT compiled code with LLVM causes random crashes since Mono 5.16 (was: "Timespan error with last version")
Steps to Reproduce
create a console application with on line:
static void Main(string[] args)
{
TimeSpan DefaultFlushTimeout = TimeSpan.FromSeconds(15);
Console.Read();
}compile and execute on Linux
- crash
Current Behavior
crash
Expected Behavior
working
On which platforms did you notice this
[ ] macOS
[ x] Linux raspberrypi 4.14.70-v7+ #1144 SMP Tue Sep 18 17:34:46 BST 2018 armv7l GNU/Linux
[ ] Windows
Version Used:
Mono JIT compiler version 5.16.0.179 (tarball Thu Oct 4 12:22:11 UTC 2018)
Stacktrace
Unhandled Exception:
System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bb8 + 0x0010c> in <c3c5f4bb011a4af8b925b0d39ee12396>:0
at System.TimeSpan.FromSeconds (System.Double value) <0x74a03e88 + 0x0001b> in <c3c5f4bb011a4af8b925b0d39ee12396>:0
at ConsoleApp1.Program.Main (System.String[] args) [0x00001] in <49d5660f8e7d40cdb5d5396ccc0c501c>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bb8 + 0x0010c> in <c3c5f4bb011a4af8b925b0d39ee12396>:0
at System.TimeSpan.FromSeconds (System.Double value) <0x74a03e88 + 0x0001b> in <c3c5f4bb011a4af8b925b0d39ee12396>:0
at ConsoleApp1.Program.Main (System.String[] args) [0x00001] in <49d5660f8e7d40cdb5d5396ccc0c501c>:0
EDIT: WORKAROUND
See suggested workaround https://github.com/mono/mono/issues/11095#issuecomment-436012372
JTrotta
All 45 comments
As quite a lot of programs are using this functionality, this is a serious issue.
Toxaris-NL
on 14 Oct 2018
This is critical. Duplicati backup stopped working on my Raspberry Pi. I tracked down the problem to TimeSpan.FromMilliseconds(Int32.MaxValue-1) which is used in the Regex class.
I think the problem (OverflowException) occurs here in the file mono/mcs/class/referencesource/mscorlib/system/timespan.cs:
C#
private static TimeSpan Interval(double value, int scale) {
if (Double.IsNaN(value))
throw new ArgumentException(Environment.GetResourceString("Arg_CannotBeNaN"));
Contract.EndContractBlock();
double tmp = value * scale;
double millis = tmp + (value >= 0? 0.5: -0.5);
if ((millis > Int64.MaxValue / TicksPerMillisecond) || (millis < Int64.MinValue / TicksPerMillisecond))
throw new OverflowException(Environment.GetResourceString("Overflow_TimeSpanTooLong"));
return new TimeSpan((long)millis * TicksPerMillisecond);
}
The code seems correct and didn't change recently. So I am not sure what exactly happens.
Another thing I noticed is, that the Exception only happens sometimes if I execute the same program again and again, see:
```bash
pi@raspberrypi:~ $ mono Main.exe
922337203685477
pi@raspberrypi:~ $ mono Main.exe
922337203685477
Unhandled Exception:
System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bc0 + 0x0010c> in <3833a6edf2074b959d3dab898627f0ac>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0x74a03cfc + 0x0001b> in <3833a6edf2074b959d3dab898627f0ac>:0
at JH.Prog.Main () [0x00010] in <00c1beb12fc34cbd912b5e0dfce5bcbb>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bc0 + 0x0010c> in <3833a6edf2074b959d3dab898627f0ac>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0x74a03cfc + 0x0001b> in <3833a6edf2074b959d3dab898627f0ac>:0
at JH.Prog.Main () [0x00010] in <00c1beb12fc34cbd912b5e0dfce5bcbb>:0
pi@raspberrypi:~ $ mono Main.exe
922337203685477
Unhandled Exception:
System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bc0 + 0x0010c> in <3833a6edf2074b959d3dab898627f0ac>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0x74a03cfc + 0x0001b> in <3833a6edf2074b959d3dab898627f0ac>:0
at JH.Prog.Main () [0x00010] in <00c1beb12fc34cbd912b5e0dfce5bcbb>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0x74a03bc0 + 0x0010c> in <3833a6edf2074b959d3dab898627f0ac>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0x74a03cfc + 0x0001b> in <3833a6edf2074b959d3dab898627f0ac>:0
```
... Also I fail trying to analyze mono with valgrind.
Hope this helps, any ideas?
timmi-on-rails
on 17 Oct 2018
If I compile the program in Linux (Raspberry) and execute it in Windows or Linux (x86) it does not fail.
Therefore I assume it is a problem with the mono runtime on raspberry pi and not a compiler issue.
Also running "mono --interpreter Main.exe" works without a problem.
timmi-on-rails
on 18 Oct 2018
@timmi-on-rails is it ARM specific? Can't reproduce it on Ubuntu 18 x64 with mono 5.16
Also what is the 922337203685477 value in your output:
pi@raspberrypi:~ $ mono Main.exe
922337203685477
EgorBo
on 23 Oct 2018
@timmi-on-rails did you run mono --interpreter Main.exe on the Raspberry Pi? Could you post the output of mono -v -v -v --trace Main.exe to gist.github.com? That would be very helpful.
My guess: Since it works in the interpreter, and Raspberry Pi doesn't do hardware floating point, it could be a problem with "softfloat" in our ARM JIT backend
lewurm
on 23 Oct 2018
@EgorBo, yes it seems ARM specific. I cannot reproduce it on Linux x86 either.
The output was just some multiplication of Timespan.TicksPerMillisecond times some large number.
It is not relevant. Sorry for not posting a clear source code example. I will do so very soon.
timmi-on-rails
on 23 Oct 2018
@lewurm Yes I did run mono --interpreter Main.exe on the Raspberry Pi for atleast 100 times and no problem occured. I am not sure it is soft float, because I am running Rasbian 9 (debian stretch mod) and the architecture is armhf (hf = hard float). Or is that something different? It is a Raspberry Pi 2.
After the problem occured I had to roll back mono to 5.14 on my Raspberry Pi to get Duplicati working again. Now I wanted to reproduce the problem on a fresh install (separate SD card) with latest mono 5.16 and I have trouble reproducing it again. I upgraded the mono version on my original server (SD card) again and couldn't reproduce it.
Maybe there is already a patch added to raspbian since I tryed last week?
I am not sure how I can figure that out...
Or maybe this is just an issue when updating from 5.14 to 5.16 with apt upgrade, maybe some precompile step is left out... I haven't tested that, because this time I removed the old version and installed the new instead of upgrading.
I could try that again. Can anybody check if there has been a debian/raspbian patch?
@JTrotta Do you use Raspbian?
timmi-on-rails
on 23 Oct 2018
@timmi-on-rails Yes: Linux raspberrypi 4.14.70-v7+ #1144 SMP Tue Sep 18 17:34:46 BST 2018 armv7l GNU/Linux
JTrotta
on 24 Oct 2018
@JTrotta Can you please remove mono completely and reinstall the latest version 5.16.0.179 again and check if the problem still exists? That would be very helpful.
Currently I assume, that something goes wrong during the apt upgrade command.
I will try to reproduce the upgrade procedure tonight. I will install 5.14.x.x snapshot and upgrade to 5.16.0.179 and we will see what happens.
timmi-on-rails
on 24 Oct 2018
@timmi-on-rails Of course! Will do today, and let you know in a while.
JTrotta
on 24 Oct 2018
I can confirm that an installation from scratch works, on Linux raspberrypi 4.14.71-v7+ #1145 SMP Fri Sep 21 15:38:35 BST 2018 armv7l GNU/Linux.
So, as supposed, the problem is related to scripts update
JTrotta
on 24 Oct 2018
@JTrotta Thank you for trying out so fast. I am still trying to reproduce the malicious upgrade.
Anyone into upgrade scripts business here? :-)
timmi-on-rails
on 24 Oct 2018
So, I can reproduce the problem.
I removed mono completely. Then I installed mono from
deb https://download.mono-project.com/repo/debian stretch/snapshots/5.14 main
and then I changed to the latest repo and ran apt upgrade
deb https://download.mono-project.com/repo/debian stretch main
This way the error is reproducible. Upgrade from 5.14.0.177 to 5.16.0.179.
@lewurm Here the gist of mono -v -v -v --trace Main.exe, although this may be not so important since it seems an apt upgrade issue.
gist: https://gist.githubusercontent.com/timmi-on-rails/6fa644ee16710ce60a148fa08776072b/raw/5ae920c634fc0ff72c9a2cc67edd24b1d5dd9c4d/mono.log
Here is the source code:
```C#
using System;
namespace Test
{
class Program
{
public static void Main()
{
var ts = TimeSpan.FromMilliseconds(1);
}
}
}
```
So, who can help fix this?
timmi-on-rails
on 24 Oct 2018
@timmi-on-rails what's the output of mono --version on the failing installation? could you also try if your example works when you run mono -O=-aot Main.exe?
Still guessing here, but it could be related to the BCL being AOT compiled during package install with LLVM. @directhex when did we start to ship an LLVM enabled build, that also does AOT compilation of the BCL?
lewurm
on 24 Oct 2018
I can reproduce on armhf. The BCL is AOT compiled with LLVM:
$ /usr/bin/mono --aot=try-llvm,temp-path=/tmp/monoaot.2Uv5HA3GF,outfile=/usr/lib/mono/aot-cache/arm/mscorlib.dll.so -O=all /usr/lib/mono/4.5/mscorlib.dll
$ objdump -d /usr/lib/mono/aot-cache/arm/mscorlib.dll.so | grep -i -A 90 'interval_double_int>:'
0019bb40 <System_TimeSpan_Interval_double_int>:
19bb40: e92d4830 push {r4, r5, fp, lr}
19bb44: e24dd018 sub sp, sp, #24
19bb48: ec432b11 vmov d1, r2, r3
19bb4c: e1a04000 mov r4, r0
19bb50: e3a026ff mov r2, #267386880 ; 0xff00000
19bb54: e3a03000 mov r3, #0
19bb58: ed8d1b04 vstr d1, [sp, #16]
19bb5c: e3822207 orr r2, r2, #1879048192 ; 0x70000000
19bb60: e59d1014 ldr r1, [sp, #20]
19bb64: e59d0010 ldr r0, [sp, #16]
19bb68: e3c11102 bic r1, r1, #-2147483648 ; 0x80000000
19bb6c: e1510002 cmp r1, r2
19bb70: 83a03001 movhi r3, #1
19bb74: e3500000 cmp r0, #0
19bb78: 13a00001 movne r0, #1
19bb7c: e1510002 cmp r1, r2
19bb80: 11a00003 movne r0, r3
19bb84: e3500000 cmp r0, #0
19bb88: 1a000023 bne 19bc1c <System_TimeSpan_Interval_double_int+0xdc>
19bb8c: eeb51bc0 vcmpe.f64 d1, #0.0
19bb90: e59d0028 ldr r0, [sp, #40] ; 0x28
19bb94: e28f10b4 add r1, pc, #180 ; 0xb4
19bb98: ed9f3b30 vldr d3, [pc, #192] ; 19bc60 <System_TimeSpan_Interval_double_int+0x120>
19bb9c: eef1fa10 vmrs APSR_nzcv, fpscr
19bba0: b2811008 addlt r1, r1, #8
19bba4: ed910b00 vldr d0, [r1]
19bba8: ee020a10 vmov s4, r0
19bbac: eeb82bc2 vcvt.f64.s32 d2, s4
19bbb0: ee020b01 vmla.f64 d0, d2, d1
19bbb4: eeb40bc3 vcmpe.f64 d0, d3
19bbb8: eef1fa10 vmrs APSR_nzcv, fpscr
19bbbc: ca00001c bgt 19bc34 <System_TimeSpan_Interval_double_int+0xf4>
19bbc0: ed9f1b28 vldr d1, [pc, #160] ; 19bc68 <System_TimeSpan_Interval_double_int+0x128>
19bbc4: eeb40bc1 vcmpe.f64 d0, d1
19bbc8: eef1fa10 vmrs APSR_nzcv, fpscr
19bbcc: 4a000018 bmi 19bc34 <System_TimeSpan_Interval_double_int+0xf4>
19bbd0: eb130bfe bl 65ebd0 <plt__jit_icall___emul_fconv_to_i8> 19bbd4: e3a02e71 mov r2, #1808 ; 0x710
19bbd8: e3a03000 mov r3, #0 19bbdc: e3a05000 mov r5, #0
19bbe0: e3822a02 orr r2, r2, #8192 ; 0x2000
19bbe4: eb12ed91 bl 657230 <plt__jit_icall___emul_lmul> 19bbe8: e58d5004 str r5, [sp, #4]
19bbec: e58d5000 str r5, [sp]
19bbf0: e58d1004 str r1, [sp, #4] 19bbf4: e58d0000 str r0, [sp]
19bbf8: e58d100c str r1, [sp, #12] 19bbfc: e58d0008 str r0, [sp, #8]
19bc00: e59d0008 ldr r0, [sp, #8]
19bc04: e5840000 str r0, [r4]
19bc08: e59d000c ldr r0, [sp, #12]
19bc0c: e5840004 str r0, [r4, #4]
19bc10: e28dd018 add sp, sp, #24
19bc14: e8bd4830 pop {r4, r5, fp, lr}
19bc18: e1a0f00e mov pc, lr
19bc1c: e59f0050 ldr r0, [pc, #80] ; 19bc74 <System_TimeSpan_Interval_double_int+0x134>
19bc20: eb12de1a bl 653490 <plt__jit_icall_mono_helper_ldstr_mscorlib>
19bc24: e1a01000 mov r1, r0
19bc28: e59f0048 ldr r0, [pc, #72] ; 19bc78 <System_TimeSpan_Interval_double_int+0x138> 19bc2c: eb12de6b bl 6535e0 <plt__jit_icall_mono_create_corlib_exception_1>
19bc30: eb12de1e bl 6534b0 <plt__jit_icall_mono_arch_throw_exception>
19bc34: e3a00032 mov r0, #50 ; 0x32
19bc38: e3800801 orr r0, r0, #65536 ; 0x10000
19bc3c: eb12de13 bl 653490 <plt__jit_icall_mono_helper_ldstr_mscorlib>
19bc40: e1a01000 mov r1, r0
19bc44: e59f0024 ldr r0, [pc, #36] ; 19bc70 <System_TimeSpan_Interval_double_int+0x130>
19bc48: eb12de64 bl 6535e0 <plt__jit_icall_mono_create_corlib_exception_1> 19bc4c: eb12de17 bl 6534b0 <plt__jit_icall_mono_arch_throw_exception> 19bc50: 00000000 .word 0x00000000 19bc54: 3fe00000 .word 0x3fe00000
19bc58: 00000000 .word 0x00000000 19bc5c: bfe00000 .word 0xbfe00000
19bc60: eb1c4328 .word 0xeb1c4328 19bc64: 430a36e2 .word 0x430a36e2
19bc68: eb1c4328 .word 0xeb1c4328
19bc6c: c30a36e2 .word 0xc30a36e2 19bc70: 020001bb .word 0x020001bb
19bc74: 000101fd .word 0x000101fd
19bc78: 02000137 .word 0x02000137
md5-e62d1f121212a93ae5167cf38ca08c3b
0012b410 <System_TimeSpan_Interval_double_int>:
12b410: e92d4830 push {r4, r5, fp, lr}
12b414: e24dd008 sub sp, sp, #8
12b418: e3a0c6ff mov ip, #267386880 ; 0xff00000
12b41c: e1a04000 mov r4, r0
12b420: e3c30102 bic r0, r3, #-2147483648 ; 0x80000000
12b424: e38cc207 orr ip, ip, #1879048192 ; 0x70000000
12b428: e2721000 rsbs r1, r2, #0
12b42c: e0dc0000 sbcs r0, ip, r0
12b430: 3a000028 bcc 12b4d8 <System_TimeSpan_Interval_double_int+0xc8>
12b434: ec432b11 vmov d1, r2, r3
12b438: e59d0018 ldr r0, [sp, #24]
12b43c: eeb51bc0 vcmpe.f64 d1, #0.0
12b440: eef1fa10 vmrs APSR_nzcv, fpscr
12b444: ee000a10 vmov s0, r0
12b448: eeb82bc0 vcvt.f64.s32 d2, s0
12b44c: e28f00a4 add r0, pc, #164 ; 0xa4
12b450: a2800008 addge r0, r0, #8
12b454: ed900b00 vldr d0, [r0]
12b458: ee020b01 vmla.f64 d0, d2, d1
12b45c: ed9f1b29 vldr d1, [pc, #164] ; 12b508 <System_TimeSpan_Interval_double_int+0xf8>
12b460: eeb40bc1 vcmpe.f64 d0, d1
12b464: eef1fa10 vmrs APSR_nzcv, fpscr
12b468: ca000011 bgt 12b4b4 <System_TimeSpan_Interval_double_int+0xa4>
12b46c: ed9f1b27 vldr d1, [pc, #156] ; 12b510 <System_TimeSpan_Interval_double_int+0x100>
12b470: eeb40bc1 vcmpe.f64 d0, d1
12b474: eef1fa10 vmrs APSR_nzcv, fpscr
12b478: 4a00000d bmi 12b4b4 <System_TimeSpan_Interval_double_int+0xa4>
12b47c: eb12c4ff bl 5dc880 <plt__jit_icall___emul_fconv_to_i8>
12b480: e3a02e71 mov r2, #1808 ; 0x710
12b484: e3a03000 mov r3, #0
12b488: e3822a02 orr r2, r2, #8192 ; 0x2000
12b48c: e3a05000 mov r5, #0
12b490: eb12bd6e bl 5daa50 <plt__jit_icall___emul_lmul>
12b494: e58d5004 str r5, [sp, #4]
12b498: e58d5000 str r5, [sp]
12b49c: e58d1004 str r1, [sp, #4]
12b4a0: e58d0000 str r0, [sp]
12b4a4: e8840003 stm r4, {r0, r1}
12b4a8: e28dd008 add sp, sp, #8
12b4ac: e8bd4830 pop {r4, r5, fp, lr}
12b4b0: e1a0f00e mov pc, lr
12b4b4: e3a00083 mov r0, #131 ; 0x83
12b4b8: e3800c8b orr r0, r0, #35584 ; 0x8b00
12b4bc: eb12b45b bl 5d8630 <plt__jit_icall_mono_helper_ldstr_mscorlib>
12b4c0: e1a01000 mov r1, r0
12b4c4: e3a000b3 mov r0, #179 ; 0xb3
12b4c8: e3800402 orr r0, r0, #33554432 ; 0x2000000
12b4cc: e2800075 add r0, r0, #117 ; 0x75
12b4d0: eb12b47a bl 5d86c0 <plt__jit_icall_mono_create_corlib_exception_1>
12b4d4: eb12b45d bl 5d8650 <plt__jit_icall_mono_arch_throw_exception>
12b4d8: e3a0004e mov r0, #78 ; 0x4e
12b4dc: e3800c8d orr r0, r0, #36096 ; 0x8d00
12b4e0: eb12b452 bl 5d8630 <plt__jit_icall_mono_helper_ldstr_mscorlib>
12b4e4: e1a01000 mov r1, r0
12b4e8: e3a000b3 mov r0, #179 ; 0xb3
12b4ec: e3800402 orr r0, r0, #33554432 ; 0x2000000
12b4f0: eb12b472 bl 5d86c0 <plt__jit_icall_mono_create_corlib_exception_1>
12b4f4: eb12b455 bl 5d8650 <plt__jit_icall_mono_arch_throw_exception>
12b4f8: 00000000 andeq r0, r0, r0
12b4fc: bfe00000 svclt 0x00e00000
12b500: 00000000 andeq r0, r0, r0
12b504: 3fe00000 svccc 0x00e00000
12b508: eb1c4328 bl 83c1b0 <__bss_end__+0x12184c>
12b50c: 430a36e2 movwmi r3, #42722 ; 0xa6e2
12b510: eb1c4328 bl 83c1b8 <__bss_end__+0x121854>
12b514: c30a36e2 movwgt r3, #42722 ; 0xa6e2
@timmi-on-rails as a workaround until it is fixed, I suggest to use -O=-aot. You can set export MONO_ENV_OPTIONS='-O=-aot' so it will picked up by all programs using mono.
lewurm
on 25 Oct 2018
@lewurm Thank you for your effort. When you say you can reproduce with the current master, do you mean the apt upgrade procedure or is there actually a bug besides the upgrade scripts?
Because for me the workaround is simply remove mono and install it again and it works.
The issue only appears if I upgrade with apt upgrade from a previous mono version e.g. 5.14 -> 5.16
Maybe the step during apt upgrade when the BCL gets AOT compiled happens too early, when not all packages are installed/ set up? I will try to AOT recompile the BCL manually and see if that solves the problem. Then this would be a great workaround, if people don't need to completely reinstall mono, because this can take up to an hour on raspberry pi 2.
timmi-on-rails
on 25 Oct 2018
@timmi-on-rails with current master I mean that I built mono myself from source, and invoking the AOT compiler for mscorlib.dll similar to what apt would do. I'm pretty sure it's a bug with our LLVM support, nothing wrong with the packaging imho (except maybe that it shouldn't ship it with LLVM 馃槢 ).
Could you try my recommended workaround export MONO_ENV_OPTIONS='-O=-aot'?
Some observation: I applied the following patch to corefx:
--- a/src/Common/src/CoreLib/System/TimeSpan.cs
+++ b/src/Common/src/CoreLib/System/TimeSpan.cs
@@ -240,6 +240,8 @@ namespace System
private static TimeSpan Interval(double value, int scale)
{
+ Console.WriteLine ("Interval arg1: " + value);
+ Console.WriteLine ("Interval arg2: " + scale);
if (Double.IsNaN(value))
throw new ArgumentException(SR.Arg_CannotBeNaN);
double tmp = value * scale;
root@xam-jetsontx1-5:/mono# cat repro.cs
using System;
namespace Test
{
class Program
{
public static void Main()
{
var ts = TimeSpan.FromMilliseconds(1);
}
}
}
root@xam-jetsontx1-5:/mono# ./mono/mini/mono-sgen -O=-aot repro.exe
Interval arg1: 1
Interval arg2: 1
root@xam-jetsontx1-5:/mono# ./mono/mini/mono-sgen repro.exe
Interval arg1: 0
Interval arg2: 1
Unhandled Exception:
System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0xf4dc5360 + 0x00134> in <ef044d2be74a4446b7f7ca39b9cf8a11>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0xf4dc54dc + 0x0001b> in <ef044d2be74a4446b7f7ca39b9cf8a11>:0
at Test.Program.Main () [0x00000] in <e5506ae5acd740c1b4ebcd6ec7c2d35c>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.OverflowException: TimeSpan overflowed because the duration is too long.
at System.TimeSpan.Interval (System.Double value, System.Int32 scale) <0xf4dc5360 + 0x00134> in <ef044d2be74a4446b7f7ca39b9cf8a11>:0
at System.TimeSpan.FromMilliseconds (System.Double value) <0xf4dc54dc + 0x0001b> in <ef044d2be74a4446b7f7ca39b9cf8a11>:0
at Test.Program.Main () [0x00000] in <e5506ae5acd740c1b4ebcd6ec7c2d35c>:0
There seems to be a problem with argument passing, looks like some ABI mismatch between LLVM and mono. It's seriously broken.
lewurm
on 25 Oct 2018
@lewurm I will check the things you said as soon as I can. Meanwhile do you have any explanation why reinstalling mono fixes the issue?
timmi-on-rails
on 25 Oct 2018
@timmi-on-rails what do you mean by "reinstalling"? As far as I understand, this problem starts to show up with version 5.16. As far as I know, that's the first version where we started to ship with LLVM support on linux. @directhex can you confirm that?
lewurm
on 25 Oct 2018
@lewurm He means that if we update MONO from 5.14 to 5.16, we encounter the problem. If we install MONO 5.16 from scratch there is no problem. So we think that the problem is due to some scripts during the update, scripts that work well during installation from scratch.
Are those scripts different?
JTrotta
on 25 Oct 2018
@lewurm yes, 5.16 is when LLVM support came in. LLVM support is _optional_ and can be disabled by uninstalling mono-llvm-support
directhex
on 25 Oct 2018
@lewurm
I mean exactly what JTrotta says. Install from scratch/uninstall-first works.
Blank Raspbian -> Mono 5.14 -> Upgrade to Mono 5.16 -> exception
Blank Raspbian -> Mono 5.16 -> works (no exception)
export MONO_ENV_OPTIONS='-O=-aot' or mono -O=-aot Main.exe solves the problem aswell.
mono --version for the failing install outputs:
Mono JIT compiler version 5.16.0.179 (tarball Thu Oct 4 12:05:24 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: normal
Notifications: epoll
Architecture: armel,vfp+hard
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(3.6.0svn-mono-/)
GC: sgen (concurrent by default)
Removing the package mono-llvm-support as @directhex suggested does not solve the problem.
timmi-on-rails
on 25 Oct 2018
Hi,
I faced the same issue on Raspberry Pi.
Is there an easy way to set a (native) breakpoint on the starting of a JIT-compiled function? (TimeSpan.Interval, TimeSpan.FromSeconds)
mono --breakonex works but the stack was corrupted at that timing.
mono --break <method> is available on ARM?
yappy
on 26 Oct 2018
@directhex would removing mono-llvm-support trigger the following?
- delete
/usr/lib/mono/aot-cache/arm/mscorlib.dll.so - rerun generation of
/usr/lib/mono/aot-cache/arm/mscorlib.dll.soby calling the AOT compiler _without_ LLVM
@yappy mono --break <method> only works with the JIT. In this case, the method is retrieved from the AOT image. You can either (1) disable AOT by using mono -O=-aot or (2) inspect the AOT image with objdump -d /usr/lib/mono/aot-cache/arm/mscorlib.dll.so
lewurm
on 29 Oct 2018
I also did an apt-get upgrade which installed mono-devel 5.16 and broke sonarr. However, following these instructions I installed the nightly version 5.21 which fixed the issue with sonarr.
https://www.mono-project.com/download/stable/#download-lin-raspbian
Note: I removed mono 5.16 before following the above instructions.
mikepaxton
on 30 Oct 2018
@lewurm re-AOTing mscorlib can be done with dpkg-reconfigure libmono-corlib4.5-cil. It is called with --aot=try-llvm, so with mono-llvm-support removed (which contains /usr/lib/libmono-llvm.so.0) it should use the regular AOT generator.
directhex
on 31 Oct 2018
export MONO_ENV_OPTIONS='-O=-aot' or mono -O=-aot Main.exe solves the problem aswell.
@timmi-on-rails is there a global file where I can set this, or ? I'm using a RPi and don't want to change all calls that are made using mono with different users.
msdos
on 1 Nov 2018
@mikepaxton I guess removing mono 5.16 did the trick and you could have installed the same version (5.16) again. I haven't had time yet to investigate the difference between an upgrade and install from scratch.
timmi-on-rails
on 3 Nov 2018
@msdos I have asked google and it said you should have a look at https://bash.cyberciti.biz/guide//etc/profile :)
timmi-on-rails
on 3 Nov 2018
I can confirm the following (Starting from a 5.14 -> 5.16 upgraded "erroneous TimeSpan" install.):
If I do apt --purge remove mono-llvm-support and then dpkg-reconfigure libmono-corlib4.5-cil the example works ---> no exception.
If I do apt install mono-llvm-support and then dpkg-reconfigure libmono-corlib4.5-cil again, the example stops working ---> exception.
Still, the question is why does an install of mono 5.16 from scratch solve the exception aswell?
Maybe the order of the installation of package mono-llvm-support and the dpkg-reconfigure step is different? This will be the next thing I will investigate if I have got the time.
timmi-on-rails
on 5 Nov 2018
Possible workarounds (so far)
You can choose one of the following known workarounds:
1. Install mono 5.16 from scratch _(Update: Do not use this workaround, instead workaround 2 is recommended.)_
Remove mono completely and reinstall the latest version (i.e. 5.16).
This is an easy workaround, but takes some time. Also you may have to install mono dependent packages again. Only choose this solution, if this is not a problem.
To me it is still unclear, why this solves the problem. _(Update: It's clear now, the install scripts will run the AOT compilation of mscorlib before the mono-llvm-support package is being unpacked. Therefore the LLVM AOT bug is not in action. When upgrading mono 5.14 to 5.16 the AOT compilation happens after the mono-llvm-support package is being unpacked -> exception)_
2. Remove llvm support
Run as root:
# apt remove mono-llvm-support
# dpkg-reconfigure libmono-corlib4.5-cil
3. Disable AOT compilation
You can disable AOT compilation via an environment variable export MONO_ENV_OPTIONS='-O=-aot'. It should be possible to do this globally in /etc/profile. This might affect the performance of your applications.
timmi-on-rails
on 5 Nov 2018
So, here is another update...
@lewurm First of all the mono-llvm-support package for armhf is there since mono 5.14. (Checking the packages at https://download.mono-project.com/repo/debian/pool/main/m/mono/)
Installing mono 5.16 from scratch (no exception)
Install 5.16 log
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
ca-certificates-mono cli-common libmono-btls-interface4.0-cil
libmono-corlib4.5-cil libmono-i18n-west4.0-cil libmono-i18n4.0-cil
libmono-posix4.0-cil libmono-security4.0-cil
libmono-system-configuration4.0-cil libmono-system-core4.0-cil
libmono-system-numerics4.0-cil libmono-system-security4.0-cil
libmono-system-xml4.0-cil libmono-system4.0-cil mono-4.0-gac mono-gac
mono-llvm-support mono-llvm-tools mono-runtime-common mono-runtime-sgen
Suggested packages:
libmono-i18n4.0-all libgamin0 xdg-utils | libgnome2-0 | konqueror
The following NEW packages will be installed:
ca-certificates-mono cli-common libmono-btls-interface4.0-cil
libmono-corlib4.5-cil libmono-i18n-west4.0-cil libmono-i18n4.0-cil
libmono-posix4.0-cil libmono-security4.0-cil
libmono-system-configuration4.0-cil libmono-system-core4.0-cil
libmono-system-numerics4.0-cil libmono-system-security4.0-cil
libmono-system-xml4.0-cil libmono-system4.0-cil mono-4.0-gac mono-gac
mono-llvm-support mono-llvm-tools mono-runtime mono-runtime-common
mono-runtime-sgen
0 upgraded, 21 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.7 MB of archives.
After this operation, 49.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] Get:1 https://download.mono-project.com/repo/debian stretch/main armhf mono-runtime-sgen armhf 5.16.0.179-0xamarin1+debian9b1 [1,367 kB]
Get:2 https://download.mono-project.com/repo/debian stretch/main armhf mono-runtime armhf 5.16.0.179-0xamarin1+debian9b1 [23.4 kB]
Get:3 https://download.mono-project.com/repo/debian stretch/main armhf libmono-corlib4.5-cil all 5.16.0.179-0xamarin1+debian9b1 [1,153 kB]
Get:4 https://download.mono-project.com/repo/debian stretch/main armhf libmono-posix4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [90.5 kB]
Get:5 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system-core4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [282 kB]
Get:6 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system-numerics4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [56.7 kB]
Get:7 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system-xml4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [812 kB]
Get:8 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system-security4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [83.2 kB]
Get:9 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system-configuration4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [63.0 kB]
Get:10 https://download.mono-project.com/repo/debian stretch/main armhf libmono-system4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [852 kB]
Get:11 https://download.mono-project.com/repo/debian stretch/main armhf libmono-security4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [126 kB]
Get:12 https://download.mono-project.com/repo/debian stretch/main armhf mono-4.0-gac all 5.16.0.179-0xamarin1+debian9b1 [164 kB]
Get:13 https://download.mono-project.com/repo/debian stretch/main armhf mono-gac all 5.16.0.179-0xamarin1+debian9b1 [27.5 kB]
Get:14 https://download.mono-project.com/repo/debian stretch/main armhf mono-runtime-common armhf 5.16.0.179-0xamarin1+debian9b1 [737 kB]
Get:15 https://download.mono-project.com/repo/debian stretch/main armhf ca-certificates-mono all 5.16.0.179-0xamarin1+debian9b1 [27.3 kB]
Get:16 https://download.mono-project.com/repo/debian stretch/main armhf cli-common all 0.9+xamarin8+debian9b1 [176 kB]
Get:17 https://download.mono-project.com/repo/debian stretch/main armhf libmono-btls-interface4.0-cil armhf 5.16.0.179-0xamarin1+debian9b1 [30.2 kB]
Get:18 https://download.mono-project.com/repo/debian stretch/main armhf libmono-i18n4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [31.7 kB]
Get:19 https://download.mono-project.com/repo/debian stretch/main armhf libmono-i18n-west4.0-cil all 5.16.0.179-0xamarin1+debian9b1 [36.6 kB]
Get:20 https://download.mono-project.com/repo/debian stretch/main armhf mono-llvm-tools armhf 3.6.0+mono201805011452-0xamarin1+debian9b1 [9,223 kB]
Get:21 https://download.mono-project.com/repo/debian stretch/main armhf mono-llvm-support armhf 5.16.0.179-0xamarin1+debian9b1 [1,322 kB]
Fetched 16.7 MB in 6s (2,408 kB/s)
Selecting previously unselected package mono-runtime-sgen.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36259 files and directories currently installed.)
Preparing to unpack .../00-mono-runtime-sgen_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime-sgen (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-runtime.
Preparing to unpack .../01-mono-runtime_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-corlib4.5-cil.
Preparing to unpack .../02-libmono-corlib4.5-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-corlib4.5-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-posix4.0-cil.
Preparing to unpack .../03-libmono-posix4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-posix4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system-core4.0-cil.
Preparing to unpack .../04-libmono-system-core4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-core4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system-numerics4.0-cil.
Preparing to unpack .../05-libmono-system-numerics4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-numerics4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system-xml4.0-cil.
Preparing to unpack .../06-libmono-system-xml4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-xml4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system-security4.0-cil.
Preparing to unpack .../07-libmono-system-security4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system-configuration4.0-cil.
Preparing to unpack .../08-libmono-system-configuration4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-configuration4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-system4.0-cil.
Preparing to unpack .../09-libmono-system4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-security4.0-cil.
Preparing to unpack .../10-libmono-security4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-4.0-gac.
Preparing to unpack .../11-mono-4.0-gac_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking mono-4.0-gac (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-gac.
Preparing to unpack .../12-mono-gac_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking mono-gac (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-runtime-common.
Preparing to unpack .../13-mono-runtime-common_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime-common (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-runtime-sgen (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-runtime (5.16.0.179-0xamarin1+debian9b1) ...
update-alternatives: using /usr/bin/mono to provide /usr/bin/cli (cli) in auto mode
Setting up libmono-corlib4.5-cil (5.16.0.179-0xamarin1+debian9b1) ...
Mono precompiling /usr/lib/mono/4.5/mscorlib.dll for arm...
Setting up libmono-system-numerics4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-configuration4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-4.0-gac (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-gac (5.16.0.179-0xamarin1+debian9b1) ...
update-alternatives: using /usr/bin/gacutil to provide /usr/bin/cli-gacutil (global-assembly-cache-tool) in auto mode
Setting up mono-runtime-common (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-xml4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-posix4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-core4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package ca-certificates-mono.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36399 files and directories currently installed.)
Preparing to unpack .../0-ca-certificates-mono_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking ca-certificates-mono (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package cli-common.
Preparing to unpack .../1-cli-common_0.9+xamarin8+debian9b1_all.deb ...
Unpacking cli-common (0.9+xamarin8+debian9b1) ...
Selecting previously unselected package libmono-btls-interface4.0-cil.
Preparing to unpack .../2-libmono-btls-interface4.0-cil_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking libmono-btls-interface4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-i18n4.0-cil.
Preparing to unpack .../3-libmono-i18n4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-i18n4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package libmono-i18n-west4.0-cil.
Preparing to unpack .../4-libmono-i18n-west4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-i18n-west4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-llvm-tools.
Preparing to unpack .../5-mono-llvm-tools_3.6.0+mono201805011452-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-llvm-tools (3.6.0+mono201805011452-0xamarin1+debian9b1) ...
Selecting previously unselected package mono-llvm-support.
Preparing to unpack .../6-mono-llvm-support_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-llvm-support (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-llvm-tools (3.6.0+mono201805011452-0xamarin1+debian9b1) ...
Processing triggers for mime-support (3.60) ...
Setting up ca-certificates-mono (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libmono-i18n4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-llvm-support (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for ca-certificates (20161130+nmu1+deb9u1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Updating Mono key store
Mono Certificate Store Sync - version 5.16.0.179
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Importing into legacy system store:
I already trust 151, your new list has 151
Import process completed.
Importing into BTLS system store:
I already trust 151, your new list has 151
Import process completed.
Done
done.
Setting up cli-common (0.9+xamarin8+debian9b1) ...
Setting up libmono-btls-interface4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-i18n-west4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Note, that the AOT compilation of mscorlib happens before the mono-llvm-support package is being unpacked. (imo the wrong order) That basically means that LLVM is not used when installing mono 5.16 from scratch. Therefore the example is not throwing an exception.
Upgrading mono 5.14 to 5.16 (exception)
Install 5.14 log
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
ca-certificates-mono cli-common libmono-btls-interface4.0-cil
libmono-corlib4.5-cil libmono-i18n-west4.0-cil libmono-i18n4.0-cil
libmono-posix4.0-cil libmono-security4.0-cil
libmono-system-configuration4.0-cil libmono-system-core4.0-cil
libmono-system-numerics4.0-cil libmono-system-security4.0-cil
libmono-system-xml4.0-cil libmono-system4.0-cil mono-4.0-gac mono-gac
mono-llvm-support mono-llvm-tools mono-runtime-common mono-runtime-sgen
Suggested packages:
libmono-i18n4.0-all libgamin0 xdg-utils | libgnome2-0 | konqueror
The following NEW packages will be installed:
ca-certificates-mono cli-common libmono-btls-interface4.0-cil
libmono-corlib4.5-cil libmono-i18n-west4.0-cil libmono-i18n4.0-cil
libmono-posix4.0-cil libmono-security4.0-cil
libmono-system-configuration4.0-cil libmono-system-core4.0-cil
libmono-system-numerics4.0-cil libmono-system-security4.0-cil
libmono-system-xml4.0-cil libmono-system4.0-cil mono-4.0-gac mono-gac
mono-llvm-support mono-llvm-tools mono-runtime mono-runtime-common
mono-runtime-sgen
0 upgraded, 21 newly installed, 0 to remove and 0 not upgraded.
Need to get 19.8 MB of archives.
After this operation, 58.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] Get:1 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-runtime-sgen armhf 5.14.0.177-0xamarin3+debian9b1 [1,345 kB]
Get:2 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-runtime armhf 5.14.0.177-0xamarin3+debian9b1 [22.6 kB]
Get:3 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-corlib4.5-cil all 5.14.0.177-0xamarin3+debian9b1 [1,111 kB]
Get:4 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-posix4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [89.6 kB]
Get:5 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system-core4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [281 kB]
Get:6 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system-numerics4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [57.6 kB]
Get:7 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system-xml4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [812 kB]
Get:8 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system-security4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [77.1 kB]
Get:9 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system-configuration4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [62.0 kB]
Get:10 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-system4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [840 kB]
Get:11 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-security4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [124 kB]
Get:12 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-4.0-gac all 5.14.0.177-0xamarin3+debian9b1 [163 kB]
Get:13 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-gac all 5.14.0.177-0xamarin3+debian9b1 [26.7 kB]
Get:14 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-runtime-common armhf 5.14.0.177-0xamarin3+debian9b1 [732 kB]
Get:15 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf ca-certificates-mono all 5.14.0.177-0xamarin3+debian9b1 [26.4 kB]
Get:16 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf cli-common all 0.9+xamarin7+debian9b1 [176 kB]
Get:17 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-btls-interface4.0-cil armhf 5.14.0.177-0xamarin3+debian9b1 [29.3 kB]
Get:18 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-i18n4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [30.9 kB]
Get:19 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf libmono-i18n-west4.0-cil all 5.14.0.177-0xamarin3+debian9b1 [35.8 kB]
Get:20 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-llvm-tools armhf 3.6.0+mono201606291653-0xamarin3+debian9b1 [9,223 kB]
Get:21 https://download.mono-project.com/repo/debian stretch/snapshots/5.14/main armhf mono-llvm-support armhf 5.14.0.177-0xamarin3+debian9b1 [4,532 kB]
Fetched 19.8 MB in 18s (1,069 kB/s)
Selecting previously unselected package mono-runtime-sgen.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36259 files and directories currently installed.)
Preparing to unpack .../00-mono-runtime-sgen_5.14.0.177-0xamarin3+debian9b1_armhf.deb ...
Unpacking mono-runtime-sgen (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-runtime.
Preparing to unpack .../01-mono-runtime_5.14.0.177-0xamarin3+debian9b1_armhf.deb ...
Unpacking mono-runtime (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-corlib4.5-cil.
Preparing to unpack .../02-libmono-corlib4.5-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-corlib4.5-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-posix4.0-cil.
Preparing to unpack .../03-libmono-posix4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-posix4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system-core4.0-cil.
Preparing to unpack .../04-libmono-system-core4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system-core4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system-numerics4.0-cil.
Preparing to unpack .../05-libmono-system-numerics4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system-numerics4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system-xml4.0-cil.
Preparing to unpack .../06-libmono-system-xml4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system-xml4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system-security4.0-cil.
Preparing to unpack .../07-libmono-system-security4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system-security4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system-configuration4.0-cil.
Preparing to unpack .../08-libmono-system-configuration4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system-configuration4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-system4.0-cil.
Preparing to unpack .../09-libmono-system4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-system4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-security4.0-cil.
Preparing to unpack .../10-libmono-security4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-security4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-4.0-gac.
Preparing to unpack .../11-mono-4.0-gac_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking mono-4.0-gac (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-gac.
Preparing to unpack .../12-mono-gac_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking mono-gac (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-runtime-common.
Preparing to unpack .../13-mono-runtime-common_5.14.0.177-0xamarin3+debian9b1_armhf.deb ...
Unpacking mono-runtime-common (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-security4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-runtime-sgen (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-runtime (5.14.0.177-0xamarin3+debian9b1) ...
update-alternatives: using /usr/bin/mono to provide /usr/bin/cli (cli) in auto mode
Setting up libmono-corlib4.5-cil (5.14.0.177-0xamarin3+debian9b1) ...
Mono precompiling /usr/lib/mono/4.5/mscorlib.dll for arm...
Setting up libmono-system-numerics4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-system-configuration4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-system4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-system-xml4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-posix4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-system-core4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-system-security4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-4.0-gac (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-gac (5.14.0.177-0xamarin3+debian9b1) ...
update-alternatives: using /usr/bin/gacutil to provide /usr/bin/cli-gacutil (global-assembly-cache-tool) in auto mode
Setting up mono-runtime-common (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package ca-certificates-mono.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36396 files and directories currently installed.)
Preparing to unpack .../0-ca-certificates-mono_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking ca-certificates-mono (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package cli-common.
Preparing to unpack .../1-cli-common_0.9+xamarin7+debian9b1_all.deb ...
Unpacking cli-common (0.9+xamarin7+debian9b1) ...
Selecting previously unselected package libmono-btls-interface4.0-cil.
Preparing to unpack .../2-libmono-btls-interface4.0-cil_5.14.0.177-0xamarin3+debian9b1_armhf.deb ...
Unpacking libmono-btls-interface4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-i18n4.0-cil.
Preparing to unpack .../3-libmono-i18n4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-i18n4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package libmono-i18n-west4.0-cil.
Preparing to unpack .../4-libmono-i18n-west4.0-cil_5.14.0.177-0xamarin3+debian9b1_all.deb ...
Unpacking libmono-i18n-west4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-llvm-tools.
Preparing to unpack .../5-mono-llvm-tools_3.6.0+mono201606291653-0xamarin3+debian9b1_armhf.deb ...
Unpacking mono-llvm-tools (3.6.0+mono201606291653-0xamarin3+debian9b1) ...
Selecting previously unselected package mono-llvm-support.
Preparing to unpack .../6-mono-llvm-support_5.14.0.177-0xamarin3+debian9b1_armhf.deb ...
Unpacking mono-llvm-support (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-llvm-tools (3.6.0+mono201606291653-0xamarin3+debian9b1) ...
Processing triggers for mime-support (3.60) ...
Setting up ca-certificates-mono (5.14.0.177-0xamarin3+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libmono-i18n4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-llvm-support (5.14.0.177-0xamarin3+debian9b1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for ca-certificates (20161130+nmu1+deb9u1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Updating Mono key store
Mono Certificate Store Sync - version 5.14.0.177
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Importing into legacy system store:
I already trust 151, your new list has 151
Import process completed.
Importing into BTLS system store:
I already trust 151, your new list has 151
Import process completed.
Done
done.
Setting up cli-common (0.9+xamarin7+debian9b1) ...
Setting up libmono-btls-interface4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-i18n-west4.0-cil (5.14.0.177-0xamarin3+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Upgrade to 5.16 log
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
ca-certificates-mono cli-common libmono-btls-interface4.0-cil
libmono-corlib4.5-cil libmono-i18n-west4.0-cil libmono-i18n4.0-cil
libmono-posix4.0-cil libmono-security4.0-cil
libmono-system-configuration4.0-cil libmono-system-core4.0-cil
libmono-system-numerics4.0-cil libmono-system-security4.0-cil
libmono-system-xml4.0-cil libmono-system4.0-cil mono-4.0-gac mono-gac
mono-llvm-support mono-llvm-tools mono-runtime mono-runtime-common
mono-runtime-sgen
21 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/16.7 MB of archives.
After this operation, 8,824 kB disk space will be freed.
Do you want to continue? [Y/n] apt-listchanges: Reading changelogs...
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36464 files and directories currently installed.)
Preparing to unpack .../00-libmono-corlib4.5-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-corlib4.5-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../01-libmono-i18n4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-i18n4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../02-libmono-system-xml4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-xml4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../03-libmono-system-configuration4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-configuration4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../04-libmono-system4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../05-mono-runtime_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../06-mono-runtime-sgen_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime-sgen (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../07-mono-llvm-tools_3.6.0+mono201805011452-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-llvm-tools (3.6.0+mono201805011452-0xamarin1+debian9b1) over (3.6.0+mono201606291653-0xamarin3+debian9b1) ...
Preparing to unpack .../08-mono-llvm-support_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-llvm-support (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../09-mono-runtime-common_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking mono-runtime-common (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../10-mono-gac_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking mono-gac (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../11-mono-4.0-gac_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking mono-4.0-gac (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../12-libmono-security4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../13-libmono-posix4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-posix4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../14-libmono-system-core4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-core4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../15-libmono-system-numerics4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-numerics4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../16-libmono-system-security4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-system-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Setting up libmono-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-runtime-sgen (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-runtime (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-corlib4.5-cil (5.16.0.179-0xamarin1+debian9b1) ...
Mono precompiling /usr/lib/mono/4.5/mscorlib.dll for arm...
Setting up libmono-system-numerics4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-configuration4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-4.0-gac (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-gac (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-runtime-common (5.16.0.179-0xamarin1+debian9b1) ...
Installing new version of config file /etc/mono/config ...
Setting up libmono-system4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-xml4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-posix4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-core4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-system-security4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 36467 files and directories currently installed.)
Preparing to unpack .../ca-certificates-mono_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking ca-certificates-mono (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../cli-common_0.9+xamarin8+debian9b1_all.deb ...
Unpacking cli-common (0.9+xamarin8+debian9b1) over (0.9+xamarin7+debian9b1) ...
Preparing to unpack .../libmono-btls-interface4.0-cil_5.16.0.179-0xamarin1+debian9b1_armhf.deb ...
Unpacking libmono-btls-interface4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Preparing to unpack .../libmono-i18n-west4.0-cil_5.16.0.179-0xamarin1+debian9b1_all.deb ...
Unpacking libmono-i18n-west4.0-cil (5.16.0.179-0xamarin1+debian9b1) over (5.14.0.177-0xamarin3+debian9b1) ...
Setting up mono-llvm-tools (3.6.0+mono201805011452-0xamarin1+debian9b1) ...
Processing triggers for mime-support (3.60) ...
Setting up ca-certificates-mono (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libmono-i18n4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up mono-llvm-support (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for ca-certificates (20161130+nmu1+deb9u1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Updating Mono key store
Mono Certificate Store Sync - version 5.16.0.179
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.
Importing into legacy system store:
I already trust 151, your new list has 151
Import process completed.
Importing into BTLS system store:
I already trust 151, your new list has 151
Import process completed.
Done
done.
Setting up cli-common (0.9+xamarin8+debian9b1) ...
Setting up libmono-btls-interface4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Setting up libmono-i18n-west4.0-cil (5.16.0.179-0xamarin1+debian9b1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
When upgrading to mono 5.16 the mono-llvm-support package is being unpacked before the AOT compilation of mscorlib happens. (imo the right order) This explains why the upgraded 5.16. version throws an exception with the example program.
Conclusion
1. LLVM AOT Bug
As @lewurm pointed out there is a bug in the LLVM enabled AOT compilation of the mscorlib.
The bug was introduced in 5.16, because I checked the 5.14 version with manually running AOT compilation of mscorlib again and there happens no exception in the example program. Hence the bug was introduced after 5.14. Should we create a new issue for investigating the LLVM AOT bug? Who can help?
2. Debian packages install/upgrade scripts issue
There is a problem when installing mono from scratch, because the mono-llvm-support package is being unpacked after the AOT compilation of mscorlib has happened. When upgrading an existing mono install, the order is the other way around. Imo the right order is to unpack the mono-llvm-support package first and then do the mscorlib AOT compilation. Also install and upgrade scripts should be consistent regarding the order of the two dependent steps. Who can help? (@directhex ?)
Should we create a new issue for this matter?
timmi-on-rails
on 6 Nov 2018
Thanks @timmi-on-rails for the instructions for working around this issue. I've edited the first post and put a reference to it, so other affected users find it quickly.
The underlying issue is indeed an ABI issue, it took me a while to figure out a proper fix: https://github.com/mono/llvm/pull/16
lewurm
on 6 Nov 2018
If you're on the preview repo, try updating the mono-llvm-tools package. The new build should include a fix for this.
directhex
on 9 Nov 2018
@lewurm fixed the LLVM bug, but what about the install/update script issue, that I described?
timmi-on-rails
on 10 Nov 2018
I'm not sure your description makes sense. Apt unpacks everything before configuring anything, so AOT will happen after LLVM is on disk (and therefore after --aot=try-llvm is okay to use), even if the mono-llvm-support package gets configured after libmono-corlib4.5-cil
One change I've made is LLVM should now only be suggested (non-default) on ARM platforms.
directhex
on 12 Nov 2018
@directhex I thought so aswell, this is why I attached the log files of the installation processes to prove that the compilation happens before everything is unpacked. (See my comment and press the black triangles to unfold the installation logs).
So is this is even an aptitude bug?
timmi-on-rails
on 14 Nov 2018
... huh. Looking at your log, I think you're right. It's due to the use of Pre-Depends in ca-certificates-mono, which forces mono-runtime-common to be installed _and configured_ before ca-certificates-mono can be unpacked. But that forces the smallest possible dependency chain for mono-runtime-common to be installed/configured before anything else starts, which means enough Mono for gacutil to run (considered the bare-minimum base set for Mono packaging), which means the corlib. So, yes, as you surmised, it _is_ happening how you thought.
_Roslyn_ is likely being LLVM-AOT'd, but corlib is classic AOT.
directhex
on 15 Nov 2018
@directhex Thank you for verifying and the detailed explanation. Should I open another issue for this?
timmi-on-rails
on 15 Nov 2018
Yeah, I guess. I'll need to consult with smarter people than me on how to fix it.
directhex
on 15 Nov 2018
Could you, @timmi-on-rails, or someone else confirm that the new package fixes this issue? Thanks
lewurm
on 19 Nov 2018
@lewurm Do I need to compile from source or are there precompiled packages available? If there are precompiled packages, then I will check it this week.
timmi-on-rails
on 19 Nov 2018
@timmi-on-rails the fix is included in the preview repository https://github.com/mono/mono/issues/11095#issuecomment-437407216
lewurm
on 19 Nov 2018
@lewurm Sorry for the delay. I tested the preview version today and it worked.
No errors, when using the mono-llvm-support. (I manually called dpkg-reconfigure libmono-corlib4.5-cil upfront to make sure the mscorlib is AOT'd with llvm.)
Tested mono version:
Mono JIT compiler version 5.18.0.209 (tarball Sat Nov 24 09:39:13 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: normal
Notifications: epoll
Architecture: armel,vfp+hard
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)
Thanks for the confirmation @timmi-on-rails. I'll close the issue 馃檪
lewurm
on 3 Dec 2018
Related issues
kdubau
路
3Comments
equalent
路
4Comments
zulhfreelancer
路
4Comments
JoseTiagoCarvalho
路
3Comments
decriptor
路
4Comments
Most helpful comment
Possible workarounds (so far)
You can choose one of the following known workarounds:
1. Install mono 5.16 from scratch _(Update: Do not use this workaround, instead workaround 2 is recommended.)_
Remove mono completely and reinstall the latest version (i.e. 5.16).
This is an easy workaround, but takes some time. Also you may have to install mono dependent packages again. Only choose this solution, if this is not a problem.
To me it is still unclear, why this solves the problem. _(Update: It's clear now, the install scripts will run the AOT compilation of mscorlib before the mono-llvm-support package is being unpacked. Therefore the LLVM AOT bug is not in action. When upgrading mono 5.14 to 5.16 the AOT compilation happens after the mono-llvm-support package is being unpacked -> exception)_
2. Remove llvm support
Run as root:
3. Disable AOT compilation
You can disable AOT compilation via an environment variable
export MONO_ENV_OPTIONS='-O=-aot'. It should be possible to do this globally in/etc/profile. This might affect the performance of your applications.