Mono: LLVM AOT caches require SSE4.1-capable CPU

[Nichtraucher]

Steps to Reproduce

1. update to 5.18.0.225-0xamarin1+ubuntu1404b1
2. 
   
   
   3.
   

Current Behavior

update fails due to unresolved dependencies and other errors (see terminal output below)

Expected Behavior

errr...update goes through smoothly (as always)?

On which platforms did you notice this

Linux (Linux Mint 17.2 Rafaela (4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux). I'm using the repositories provided by mono.

Version Used:
Mono JIT compiler version 5.18.0.225
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)

Stacktrace

Terminal output here: https://pastebin.com/um5HptcP

I read in the chat that at least one other user seems to have the same/a similar issue, she posted her output here: https://0x0.st/sdBY.txt , hence my decision to post this report

any help is much appreciated!

[guydavis]

Confirmed same problem with 5.18.0.225-0xamarin1+ubuntu1604b1

Workaround for Ubuntu 16.04 is to roll back to older 5.16 version which still works:

$ sudo apt remove mono* libmono*

Edit /etc/apt/sources.list.d/mono-official-stable.list to comment out stable and pin older version:
    #deb http://download.mono-project.com/repo/ubuntu stable-xenial main
    deb http://download.mono-project.com/repo/ubuntu xenial/snapshots/5.16.0.220 main

$ sudo apt update
$ sudo apt install mono-devel

Then was able to run Jackett, Sonarr, and Radarr again. I'll leave Mono pinned to 5.16 until this issue is resolved.

[simeon4110]

I get stuck in a SIGKILL when attempting to remove the package:

=================================================================
Got a SIGILL while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)
E: installing Assembly /usr/lib/cli/nunit.core.interfaces-2.6.3/nunit.core.interfaces.dll failed
E: Installation of libnunit-core-interfaces2.6.3-cil with /usr/share/cli-common/runtimes.d/mono failed
* Installing 1 assembly from libnunit-framework2.6.3-cil into Mono
Stacktrace:

/proc/self/maps:
40900000-40920000 rwxp 00000000 00:00 0
41322000-41332000 rwxp 00000000 00:00 0
55da82ca9000-55da830f2000 r-xp 00000000 08:22 13895047                   /usr/bin/mono-sgen
55da832f1000-55da832f8000 r--p 00448000 08:22 13895047                   /usr/bin/mono-sgen
55da832f8000-55da832fd000 rw-p 0044f000 08:22 13895047                   /usr/bin/mono-sgen
55da832fd000-55da8338e000 rw-p 00000000 00:00 0
55da84d2f000-55da84f9b000 rw-p 00000000 00:00 0                          [heap]
7f5ee0000000-7f5ee0021000 rw-p 00000000 00:00 0
7f5ee0021000-7f5ee4000000 ---p 00000000 00:00 0
7f5ee5c00000-7f5ee5d00000 rw-p 00000000 00:00 0
7f5ee5dc0000-7f5ee5dc1000 ---p 00000000 00:00 0
7f5ee5dc1000-7f5ee5dc2000 rw-p 00000000 00:00 0
7f5ee5dc2000-7f5ee5dca000 ---p 00000000 00:00 0
7f5ee5dca000-7f5ee5fc1000 rw-p 00000000 00:00 0
7f5ee5fc1000-7f5ee657d000 r-xp 00000000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee657d000-7f5ee677c000 ---p 005bc000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677c000-7f5ee677d000 r--p 005bb000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677d000-7f5ee677e000 rw-p 005bc000 08:22 14290069                   /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f5ee677e000-7f5ee67a8000 rw-p 00000000 00:00 0
7f5ee67a8000-7f5ee6bff000 r--p 00000000 08:22 14290071                   /usr/lib/mono/4.5/mscorlib.dll
7f5ee6bff000-7f5ee7bff000 rw-p 00000000 00:00 0
7f5ee7bff000-7f5ee7c00000 ---p 00000000 00:00 0
7f5ee7c00000-7f5ee8800000 rw-p 00000000 00:00 0
7f5ee885d000-7f5ee88aa000 r--p 00000000 08:22 14290003                   /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7f5ee88aa000-7f5ee8b88000 r--p 00000000 08:22 13896927                   /usr/lib/locale/locale-archive
Memory around native instruction pointer (0x7f5ee60e48cf):
0x7f5ee60e48bf  20 48 09 d6 41 0f b6 53 05 48 c1 e2 28 48 09 f2   H..A..S.H..(H..
0x7f5ee60e48cf  66 41 0f 38 32 4b 06 66 0f 6f 05 72 75 49 00 66  fA.82K.f.o.ruI.f
0x7f5ee60e48df  0f 38 00 c8 66 48 0f 7e c8 66 48 0f 3a 16 ce 01  .8..fH.~.fH.:...
0x7f5ee60e48ef  48 09 c6 48 09 d6 41 0f b6 c2 41 0f b6 53 09 48  H..H..A...A..S.H

Native stacktrace:

        /usr/bin/mono(+0x129f8d) [0x55da82dd2f8d]
        /usr/bin/mono(+0x12a295) [0x55da82dd3295]
        /usr/bin/mono(+0xbfd2f) [0x55da82d68d2f]
        /usr/bin/mono(+0x414e8) [0x55da82cea4e8]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f5ee91a3890]
        /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so(+0x1238cf) [0x7f5ee60e48cf]

Waiting for dumping threads to resume

Debug info from gdb:

[New LWP 17327]
[New LWP 17328]
[Thread debugging using libthread_db enabled]

This is a never ending loop.

[simeon4110]

I managed to get it to purge by manually removing each package with dpkg.

[Thf772]

Same problem on Ubuntu Bionic, a simple sudo apt-get upgrade is enough to reproduce the bug. The culprit is ca-certificates-mono=5.18.0.225-0xamarin1+ubuntu1804b1

Moreover, apt-get won't remove Mono automatically. It suggests apt --fix-broken install, which I tried and causes a near-endless loop of SIGILLs like @simeon4110 . Also, I can't install or upgrade any package at all, even if they are not related to Mono.

The following packages have unmet dependencies:
 ca-certificates-mono : PreDepends: mono-runtime-common but it is not going to be installed
                        Depends: mono-runtime (>= 3.0~) but it is not going to be installed
                        Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                        Depends: libmono-security4.0-cil (>= 3.0.6) but it is not going to be installed
 libglade2.0-cil : Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
 libglib2.0-cil : Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                  Depends: libmono-system-core4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
                  Depends: libmono-system4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
 libgtk2.0-cil : Depends: libmono-cairo4.0-cil (>= 3.2.1) but it is not going to be installed
                 Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
                 Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
 libnunit-console-runner2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                                    Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-core-interfaces2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                                     Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-core2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-framework2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                               Depends: libmono-system-xml4.0-cil (>= 4.2.0) but it is not going to be installed
                               Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 libnunit-mocks2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
 libnunit-util2.6.3-cil : Depends: libmono-corlib4.5-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system-configuration4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
                          Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
                          Depends: libmono-system-runtime4.0-cil (>= 2.10.1) but it is not going to be installed
                          Depends: libmono-system-xml4.0-cil (>= 4.2.0) but it is not going to be installed
                          Depends: libmono-system4.0-cil (>= 4.2.0) but it is not going to be installed
 msbuild : Depends: mono-runtime (>= 3.0~) but it is not going to be installed
           Depends: libmono-corlib4.5-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-microsoft-build-framework4.0-cil (>= 3.6.0) but it is not going to be installed
           Depends: libmono-microsoft-csharp4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-componentmodel-composition4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-configuration4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-core4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-data4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-drawing4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-identitymodel4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-io-compression-filesystem4.0-cil (>= 3.2.1) but it is not going to be installed
           Depends: libmono-system-io-compression4.0-cil (>= 3.2.1) but it is not going to be installed
           Depends: libmono-system-net-http4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-numerics4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-reflection-context4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-runtime-serialization4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-system-security4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-servicemodel4.0a-cil (>= 3.2.3) but it is not going to be installed
           Depends: libmono-system-windows-forms4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-xaml4.0-cil (>= 1.0) but it is not going to be installed
           Depends: libmono-system-xml-linq4.0-cil (>= 3.0.6) but it is not going to be installed
           Depends: libmono-system-xml4.0-cil (>= 3.12.0) but it is not going to be installed
           Depends: libmono-system4.0-cil (>= 4.0.0~alpha1) but it is not going to be installed
           Depends: libmono-windowsbase4.0-cil (>= 3.0.6) but it is not going to be installed
 referenceassemblies-pcl : Depends: mono-xbuild but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

[simeon4110]

If you purge manually it will work.

[lightway82]

I can`t install and update other packages!

How I can solve this problem?

For any action this error occurs

mono-gac (5.18.0.225-0xamarin1+ubuntu1804b1) …

• Installing 1 assembly from libglade2.0-cil into Mono
  Stacktrace:

/proc/self/maps:
408c0000-408e0000 rwxp 00000000 00:00 0
40d86000-40d96000 rwxp 00000000 00:00 0
5585b9373000-5585b97bc000 r-xp 00000000 08:06 529674 /usr/bin/mono-sgen
5585b99bb000-5585b99c2000 r--p 00448000 08:06 529674 /usr/bin/mono-sgen
5585b99c2000-5585b99c7000 rw-p 0044f000 08:06 529674 /usr/bin/mono-sgen
5585b99c7000-5585b9a58000 rw-p 00000000 00:00 0
5585bba2c000-5585bbc99000 rw-p 00000000 00:00 0 [heap]
7f92cc000000-7f92cc021000 rw-p 00000000 00:00 0
7f92cc021000-7f92d0000000 ---p 00000000 00:00 0
7f92d0600000-7f92d0700000 rw-p 00000000 00:00 0
7f92d07c1000-7f92d0d7d000 r-xp 00000000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0d7d000-7f92d0f7c000 ---p 005bc000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7c000-7f92d0f7d000 r--p 005bb000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7d000-7f92d0f7e000 rw-p 005bc000 08:06 787004 /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f92d0f7e000-7f92d0fa8000 rw-p 00000000 00:00 0
7f92d0fa8000-7f92d13ff000 r--p 00000000 08:06 529541 /usr/lib/mono/4.5/mscorlib.dll
7f92d13ff000-7f92d23ff000 rw-p 00000000 00:00 0
7f92d23ff000-7f92d2400000 ---p 00000000 00:00 0
7f92d2400000-7f92d3000000 rw-p 00000000 00:00 0
7f92d3091000-7f92d30de000 r--p 00000000 08:06 655588 /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7f92d30de000-7f92d30df000 ---p 00000000 00:00 0
7f92d30df000-7f92d30e0000 rw-p 00000000 00:00 0
7f92d30e0000-7f92d30e8000 ---p 00000000 00:00 0
7f92d30e8000-7f92d3313000 rw-p 00000000 00:00 0
7f92d3313000-7f92d3723000 r--p 00000000 08:06 525494 /usr/lib/locale/locale-archive
Memory around native instruction pointer (0x7f92d08e48cf):
0x7f92d08e48bf 20 48 09 d6 41 0f b6 53 05 48 c1 e2 28 48 09 f2 H..A..S.H..(H..
0x7f92d08e48cf 66 41 0f 38 32 4b 06 66 0f 6f 05 72 75 49 00 66 fA.82K.f.o.ruI.f
0x7f92d08e48df 0f 38 00 c8 66 48 0f 7e c8 66 48 0f 3a 16 ce 01 .8..fH.~.fH.:...
0x7f92d08e48ef 48 09 c6 48 09 d6 41 0f b6 c2 41 0f b6 53 09 48 H..H..A...A..S.H

Native stacktrace:

/usr/bin/mono(+0x129f8d) [0x5585b949cf8d]
/usr/bin/mono(+0x12a295) [0x5585b949d295]
/usr/bin/mono(+0xbfd2f) [0x5585b9432d2f]
/usr/bin/mono(+0x414e8) [0x5585b93b44e8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f92d3d3e890]
/usr/lib/mono/aot-cache/amd64/mscorlib.dll.so(+0x1238cf) [0x7f92d08e48cf]

[trolley]

I have the same result as Thf772.

[noabody]

Ubuntu 18.04:
I have multiple computers and all updated except older Athlon 9850BE which is notable because it doesn't have SSE 4.1, only 4a. Since all six computers with SSE4 updated cleanly and the two without, failed, I wondered if mono now requires that instruction set.

And yes, I do have 8 computers running Ubuntu 18.04.

[cdanderson]

If you purge manually it will work.

Is this what you meant @simeon4110?

sudo dpkg --remove --force-remove-reinstreq ca-certificates-mono

Unfortunately that didn't work and I received the following error:

dpkg: dependency problems prevent removal of ca-certificates-mono:
 mono-devel depends on ca-certificates-mono (= 5.18.0.225-0xamarin1+ubuntu1804b1); however:
  Package ca-certificates-mono is to be removed.

dpkg: error processing package ca-certificates-mono (--remove):
 dependency problems - not removing
Errors were encountered while processing:
 ca-certificates-mono

Can I ask you to please be more specific with how this can be manually purged? Thank you!

[lightway82]

I decided this in a couple of hours by reinstalling the system. It turned out faster and easier.
After that, I build the library from source and use it without installation.

[noabody]

The noted regex work on Ubuntu 18.04 and I recommend not removing the PPA from software lists because that will hinder some of these. Once removal is satisfactorily complete, then remove the PPA.

Identify installed packages (dpkg):
echo $(dpkg -l '*mono*' | grep -Pio '(?<=^ii  )(mono|libmono)-.+?(?=\s.*)' | sort)

Simulate purge:
apt -s purge $(dpkg -l '*mono*' | grep -Pio '(?<=^ii  )(mono|libmono)-.+?(?=\s.*)' | sort)

Purge orphaned configs:
sudo dpkg --purge $(dpkg -l '*mono*' | grep -Pio '(?<=^rc  ).+?(?=\s.*)')

Identify PPA  and pre-format for command (ppa-purge):
find /var/lib/apt/lists -maxdepth 1 -type f -iname '*_*packages' -printf '%P\n' | perl -pe 's/(.+?)_(.+?)_(.+?)_.*/-s \1 -o \2 -p \3/gi' | grep -Pio '^((?!dists).)*$' | sort | uniq

Identify PPA packages from package list (definitive):
echo $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*mono-project*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \; | sort | uniq)

Identify PPA installed packages:
echo $(apt -s purge $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*mono-project*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \;) | grep -Pio '(?<=^purg ).+?\s' | sort | uniq)

Hold PPA installed packages:
sudo apt-mark hold $(apt -s purge $(find /var/lib/apt/lists -maxdepth 1 -type f -iname '*winehq*_*packages' -exec grep -Pio '(?<=^package: ).*' "{}" \;) | grep -Pio '(?<=^purg ).+?\s' | sort | uniq)

The grep might be helpful to identify core packages like these:

mono-gac mono4.0-gac mono-devel mono-complete mono-dbg ca-certificates-mono mono-xsp4 libmono-addins-msbuild-cil-dev libmono-addins-gui-cil-dev libmono-addins-cil-dev libmono-reflection-cil  libmono-addins-msbuild0.2-cil libmono-zeroconf-cil-dev

sudo apt removing the core will cascade removal of dependent packages then it's a matter of installing stock mono. Checking dpkg for leftover configs could help:

/usr/share/cli-common/policies.d/
libglade2.0-cil libglib2.0-cil libgtk2.0-cil libnunit-console-runner2.6.3-cil libnunit-core2.6.3-cil libnunit-core-interfaces2.6.3-cil libnunit-framework2.6.3-cil libnunit-mocks2.6.3-cil libnunit-util2.6.3-cil

/usr/lib/cli/
glade-sharp-2.0 glib-sharp-2.0 gtk-sharp-2.0 Mono.Addins-0.2 Mono.Addins.Gui-0.2 nunit-console-runner-2.6.3 nunit.core-2.6.3 nunit.core.interfaces-2.6.3 nunit.framework-2.6.3 nunit.mocks-2.6.3 nunit.util-2.6.3

(leave alone unless complete purge)
/usr/lib/mono/gac
/etc/mono

There were crash-logs in the folders mentioned above that should be easy to purge manually with a find command.

[simeon4110]

Do you want a shell script to run the procedure for you? (Not sarcastic, it takes less time to do it than explain it. )

Sent from ProtonMail mobile

-------- Original Message --------
On Dec 27, 2018, 18:58, Christopher Anderson wrote:

If you purge manually it will work.

Is this what you meant?

sudo dpkg --remove --force-remove-reinstreq ca-certificates-mono

Unfortunately that didn't work and I received the following error:

`dpkg: dependency problems prevent removal of ca-certificates-mono:
mono-devel depends on ca-certificates-mono (= 5.18.0.225-0xamarin1+ubuntu1804b1); however:
Package ca-certificates-mono is to be removed.

dpkg: error processing package ca-certificates-mono (--remove):
dependency problems - not removing
Errors were encountered while processing:
ca-certificates-mono
`

Can I ask you to be more specific with how this can be manually purged?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[noabody]

A script would be nice. Thing is, we could use ppa-purge to some extent. Have to dig into the script to see how it orders options and it doesn't cleanly remove the PPA entries in /etc/apt/sources.list

The bigger issue is that mono compiles certain components upon install. That's where my non-SSE4 processor failed to create the necessary 'exe' that in turn failed to complete the installation, leaving it broken.

Once that happens, there'll be warnings about non-empty folders (/usr/share/cli-common/policies.d /usr/lib/cli /usr/lib/mono/gac /etc/mono) and, upon purge, the user will not know if it is safe to remove them.

Sampling of commands to both install ppa and purge:

sudo apt-add-repository 'deb https://download.mono-project.com/repo/ubuntu stable-bionic main'
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo ppa-purge -s download.mono-project.com -o repo -p ubuntu

[Nabile-Rahmani]

@noabody I seriously hope this new SSE requirement is a mistake because I just updated Mono on my 2011 Debian 9 system and I'm also getting these SIGILLs.
I'm using it as a build server so that sucks. Heck, I even dev on it from time to time. Surely that's not old enough hardware to get dropped, right ? Aren't there fallbacks for compatibility ?

I'm also not gonna tempt updating and possibly breaking Mono on my main (more recent) development machine until this is figured out :s

[Nichtraucher]

Perhaps a developer can chime in and clear this up, please?

[simeon4110]

@noabody

This script will purge mono and install a stable version. I have tested it with Ubuntu 18.04, but it should work with anything after 16.04. It may seem like it gets caught in an escape loop, just let it keep running until it gets the job done.

USE AT YOUR OWN RISK: There is no error checking done with this script.

You will need to install any non devel packages once it finishes.

#!/bin/bash

# Function removes all old versions of mono, swaps out mono release repo for a 
# stable repo, and installs the stable version.
revert_to_old_mono() {
  apt remove mono* libmono-*

  # Change apt repo URL.
  mv /etc/apt/sources.list.d/mono-official.list /etc/apt/sources.list.d/mono-offical.list.old
  touch /etc/apt/sources.list.d/mono-official.list
  echo "deb http://download.mono-project.com/repo/ubuntu bionic/snapshots/5.16.0.220 main" | tee /etc/apt/sources.list.d/mono-official.list

  # Chown and chmod the new file.
  chown root:root /etc/apt/sources.list.d/mono-official.list
  chmod 0644 /etc/apt/sources.list.d/mono-official.list

  # Update and install mono version.
  apt update
  apt install mono-devel -y
}

# Reverses changes from revert_to_old_mono.
revert_to_main_repo_mono() {
  rm /etc/apt/sources.list.d/mono-official.list
  mv /etc/apt/sources.list.d/mono-offical.list.old /etc/apt/sources.list.d/mono-official.list

  apt update && apt upgrade -y
}

#### main
case "$1" in
  -f|--run-fix )
  revert_to_old_mono ;;
  -r|--revert-changes )
  revert_to_main_repo_mono ;;
  * )
  echo ""
  echo "Enter -f to run the fix or -r to revert the changes."
  echo "" ;;
esac

To run the fix, run this script with the '-f' flag.

Once the devs write a patch (I'd offer to help if I could write C#) run the script with the '-r' flag to roll back the changes to apt.

[marek-safar]

@directhex could you look into this?

[directhex]

It looks like @noabody figured it out - we hardcode SSE4.1 in LLVM AOT:

https://github.com/mono/mono/blob/40ec59d65b22590c6a5f4ed065d9f0a04de3db39/mono/mini/aot-compiler.c#L1091-L1099

This can be worked around much more easily by not installing the package mono-llvm-support (or removing it if already installed, with dpkg --purge mono-llvm-support)

@marek-safar We should probably discuss the right approach to take here - should that hardcoded instruction set assumption remain? Should LLVM AOT remain the default (over classic AOT)? Bear in mind I can make changes to the Linux packages which are distinct from changes to the main mono.git

The problem has likely existed for a few versions, going by the git commits, but due to a packaging bug I haven't bothered to fix, the first time Mono is installed, Corlib isn't LLVM AOT'd - it's an issue on upgrade or reinstall, so it's been masked for a long time.

[directhex]

OK, I've rewritten the postinst script to specifically check for SSE4.1, and always disable LLVM AOT if it's not available. I'm doing an emergency build with the fix into the vs repo of 5.16.0.220, and into preview of 5.18.0.225 - if/when that succeeds I'll promote the 5.18 build into the stable branch

[directhex]

Postinst also now gives more detail on its behaviour - i.e. it'll say if it's trying with LLVM to give users more clarity. Once the fixed packages land, users will be able to resolve the problem with an update.

[pracplayopen]

having same issue on ubuntu 16.04 systems. could make sense if llvm update was involved, successfully removing mono-llvm doesn't fix that both mono processes and mono updates fail w/the mono_crash dump loop others experiencing above..

like other users here, trying to purge the mono-certificates package failed. never had an issue of this severity before with non-edge update.

not 100% sure what changes to make w/regard to running simeon4110 's 18.04 uninstall script, suppose will look into that.

[pracplayopen]

ok had to run it multiple times to get the uninstall to complete (because of so many dumps), but this has modifications to @simeon4110 to restore mono (and could work w/other dists if you change value of dist). thx

#!/bin/bash
#dist=bionic/snapshots/5.16.0.220
#dist="stable-xenial"
dist="stable-xenial/snapshots/5.16.0.220"
#distlistpath=/etc/apt/sources.list.d/mono-official.list
distlistpath=/etc/apt/sources.list.d/mono-official-stable.list 
# Function removes all old versions of mono, swaps out mono release repo for a 
# stable repo, and installs the stable version.
revert_to_old_mono() {
  apt remove mono* libmono-*

  # Change apt repo URL.
  mv $distlistpath "$distlistpath.old"
  touch $distlistpath
  echo "deb http://download.mono-project.com/repo/ubuntu $dist main" | tee $distlistpath

  # Chown and chmod the new file.
  chown root:root $distlistpath
  chmod 0644 $distlistpath

  # Update and install mono version.
  apt update
  apt install mono-devel -y
}

# Reverses changes from revert_to_old_mono.
revert_to_main_repo_mono() {
  rm $distlistpath
  mv "$distlistpath.old" $distlistpath

  apt update && apt upgrade -y
}

#### main
case "$1" in
  -f|--run-fix )
  revert_to_old_mono ;;
  -r|--revert-changes )
  revert_to_main_repo_mono ;;
  * )
  echo ""
  echo "Enter -f to run the fix or -r to revert the changes."
  echo "" ;;
esac

[ghost]

I experienced the same issue when updating to 5.18. I was still able to update to it though.

The update failed but, just to see what would happen I then tried a complete removal (via a combination of aptitude and dpkg) and a full clean install to 5.18. Which did take.

However, something did break and has remained broken ever since -- I can no longer compile anything. For the very same reason as the OP's report. Errors in either the runtime or in native code. In this case, it's a SIGILL. And this is the case for any project, any amount of code. No matter how simple, it will not compile.

I suspect it's the very same problem deep down because the update that originally failed failed on the precompile step of some of the components.

Just thought I'd mention that it's not just updating that is failing. Using 5.18 is also less than worry free.

[lewurm]

Another workaround is to export MONO_ENV_OPTIONS=-O=-aot. That tells the runtime to always avoid AOT'd code and fall back to the JIT.

[ghost]

Can confirm that does work indeed. So, that does suggest we've found the culprit indeed. Good to know. Looking forward to seeing that remedied forthwith with a patch.

[noabody]

Intel® Software Development Emulator Download

Deailed information:

Apparently emulates SSE 4.1. As it was when SSE2/SSE3 became requirements for much of the software in use today, so too will it be with SSE4.1. Those of us with older, usable machines, know how this plays out.

[ghost]

Emulating SSE4.1 is no solution though, if I may be so bold.

• It is not going to be adopted widespread, emulation of SSE4.1. Can't very well expect people to install external tools just to continue using the one that broke in the first place. That's just messy.
• Emulating it is probably slower than not using it at all. OK, gross assumption but probably not untrue.
• Implementing a conditional in Mono is cleaner. Makes me wonder why one did not exist before release of 5.18.

For the time being, @lewurm's workaround of simply disabling the AOT optimization is the cleanest workaround (removing the mono llvm package can and probably will lead to dependency issues) until a conditional is implemented, to check for SSE4.1 support in the hardware.

[directhex]

The issue should be mitigated in 5.18.0.225-0xamarin13 and 5.16.0.220-0xamarin5

If you have existing SSE4.1-requiring AOT caches on disk, you can safely just delete the contents of /usr/lib/mono/aot-cache/

[directhex]

Realizing my change means you can't LLVM AOT on ARM any more, since ARM (surprisingly enough) fails the "does it have SSE4.1" check. Oopsie.

[Nabile-Rahmani]

Implementing a conditional in Mono is cleaner. Makes me wonder why one did not exist before release of 5.18.

Can't the CPU capabilities be probed via CPUID and the LLVM SSE4.1 optimisation added conditionally with said capability listing ?

[directhex]

Implementing a conditional in Mono is cleaner. Makes me wonder why one did not exist before release of 5.18.

Can't the CPU capabilities be probed via CPUID and the LLVM SSE4.1 optimisation added conditionally with said capability listing ?

Probably! It just... wasn't done that way.

What we're doing is just passing command-line arguments to llc, and unhelpfully, llc doesn't have anything like gcc's -march=native to optimize for the detected CPU.

[Nabile-Rahmani]

mattr, along with mcpu seem like they're defaulting to optimising for the automatically detected CPU, but I have no idea if it actually does this (or if it's even in the version in use by Mono).

But then again, the defaults for the chip might not use all the chip's features in the first place ? Who knows, though that'd be a weird choice :thinking:

[lewurm]

let's keep it open until we have propagated the fix to release branches.

[directhex]

Before making changes which affect everyone (given how badly it went last time), I'd like to enlist help from community people, especially with older CPUs, to ensure the default works as expected.

[sbrl]

@directhex: Given an archive (e.g. .tar.gz, .zip) containing a compiled binary to test, I can do so on the following CPUs that do not have SSE4.1:

Raspberry Pi 3B+

Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model:                 4
Model name:            ARMv7 Processor rev 4 (v7l)
CPU max MHz:           1200.0000
CPU min MHz:           600.0000
BogoMIPS:              38.40
Flags:                 half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

Intel Atom N2800

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               54
Model name:          Intel(R) Atom(TM) CPU N2800   @ 1.86GHz
Stepping:            1
CPU MHz:             799.833
CPU max MHz:         1862.0000
CPU min MHz:         798.0000
BogoMIPS:            3732.99
L1d cache:           24K
L1i cache:           32K
L2 cache:            512K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl nonstop_tsc cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm arat

[jaykrell]

There's no great solution to any of this.
People do various things.

• compile .dll/.so multiple times and have runtime loader pick best, possibly fallback to JIT if none of the images work on current machine
  
  
  
  • compile some functions multiple times and make a runtime choice -- but which functions?
  
  
• target low common denominators, giving up perf on most machines - if filesystem was more cloud-based, then targeting several processors and probing would be more practical

[jaykrell]

-march=native has major downside => “it works on my machine”

[Nabile-Rahmani]

@jaykrell Well in this case the Mono installation script targets the user's machine, so it has to generate code optimised and supported for that specific machine only. Hence relying on LLVM auto-detection.
Seeing as the fallback AOT has worked fine on all hardware since forever, I am inclined to think SSE4.1 isn't actually a requirement to generate AOT code with LLVM.

After waiting for GitHub to display blames, I ended up looking at the commit that introduced the requirement: https://github.com/mono/mono/commit/628aa55e74dc88d623136861dbff6a5f3af0034d#diff-499ab2170cdda019ebc31342389233b3L670

Could @vargaz clarify if this flag was added just for testing purposes or if it was vital for LLVM to even work ?

[archanox]

It appears that the SSE4.1 requirement is still in the installation scripts.
Was it determined that it was a requirement for mono's AOT?
Otherwise, this is currently breaking AOT for ARM64.

@vargaz did you have any insight?

[MattL0]

+1 @vargaz

[MattL0]

Up

[SamMonoRT]

@imhameed to follow up with @vargaz