Xamarin-android: VS2019 Crashes Android when AOT and LLVM enabled

[ianvink]

In RELEASE mode for an Android app, if we set AOT and LLVM, then build the tool on our Build server (Mac VS2019) the app crashes with the crash log attached

When we disable LLVM, it works fine

Steps to Reproduce

1. Enable AOT and LLVM
2. Build to releasr
3. Crash

Expected Behavior

Not crash

Actual Behavior

Crash

Basic Information

• Version with issue:
• Last known good version:
• IDE: VS2019
• Platform Target Frameworks:
  
  
  
  • Android: VS2019
  
  

Screenshots

Reproduction Link

log.txt

[samhouts]

/cc @jonpryor

[jonpryor]

/cc @marek-safar

[EP01]

Seeing same issue here with VS2019 on Windows, I'm assuming they use the same version of Xamarin.Android.

[agussaputrasijabat]

Same issue with VS2019 Enterprise on Windows.

[marek-safar]

Is there a way for us to repro the issue locally?

/cc @vargaz @alexanderkyte

[EP01]

I also ran into Issue 2920 in the Xamarin.Android repo, and the solution for that also appears to have resolved this issue. For those who cannot wait for a new version of Xamarin.Android to be included in Visual Studio, perhaps try the RC version detailed here.

[baskren]

Same issue here with VisualStudio Mac

[johnthiriet]

Problem still present on my side with the latest stable release on Xamarin.Android MacOS.

Edit: Just tried with the latest preview and it seems to be working. I am not comfortable enough to use it on build machines though.
Is there any estimation on when it will be added to the stable channel ?

[GiampaoloGabba]

Same issue here. There is a strange behaviour:
If i start the app in debug mode from VisualStudio, it works until i stop the debug.
Then if i reopen manually the same app in the device, it crash.

Disabling LLVM everything works fine.

I'm using Visual Studio 2019 16.0.2 on windows

[ianvink]

I get it on 2019.0.2 with:

Xamarin.Android Reference Assemblies and MSBuild support.
Mono: mono/mono/2018-08-rc@5ad371dab1b
Java.Interop: xamarin/java.interop/d16-0@c987483
LibZipSharp: grendello/LibZipSharp/master@44de300
LibZip: nih-at/libzip/rel-1-5-1@b95cf3f
MXE: xamarin/mxe/xamarin@b9cbb535
ProGuard: xamarin/proguard/master@905836d
SQLite: xamarin/sqlite/3.26.0@325e91a
Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-0@0a7edd6

[zoli13]

Same issue with VS2019 (16.0.2) + Android Release build.
If LLVM is True, it crashes right upon startup of the App. Same callstack as the issue creator.
Disabling LLVM fixes the issue.
I did not try in Debug build, as I use AOT+LLVM only in Release.

[orestesgaolin]

Same issue with VS4Mac 2019 and

Xamarin.Android
Version: 9.2.3.0 (Visual Studio Enterprise)
SDK Tools Version: 26.1.1
SDK Platform Tools Version: 28.0.3
SDK Build Tools Version: 28.0.3

After updating CI environment to newest Xamarin.Android our Android apps with enabled LLVM started to crash on startup.

[marek-safar]

@SamMonoRT could you check on this?

[SamMonoRT]

@alexanderkyte is taking a look and will report back here.

[alexanderkyte]

Since @orestesgaolin saw it on VS4M, I'll start by trying to reproduce it on OSX.

[alexanderkyte]

With VS4M 2019:

05-28 15:33:00.324  4844  4844 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 4844 (name.xacrashapp)
05-28 15:33:00.393   694   694 W         : debuggerd: handling request: pid=4844 uid=10085 gid=10085 tid=4844
05-28 15:33:00.869  4861  4861 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
05-28 15:33:00.870  4861  4861 F DEBUG   : Build fingerprint: 'Android/sdk_google_phone_armv7/generic:7.1.1/NYC/4414732:userdebug/test-keys'
05-28 15:33:00.870  4861  4861 F DEBUG   : Revision: '0'
05-28 15:33:00.871  4861  4861 F DEBUG   : ABI: 'arm'
05-28 15:33:00.872  4861  4861 F DEBUG   : pid: 4844, tid: 4844, name: name.xacrashapp  >>> com.companyname.xacrashapp <<<
05-28 15:33:00.872  4861  4861 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
05-28 15:33:00.873  4861  4861 F DEBUG   :     r0 90809a70  r1 00000000  r2 9cce9230  r3 9ebfdc00
05-28 15:33:00.874  4861  4861 F DEBUG   :     r4 90809750  r5 90809748  r6 00000000  r7 90000000
05-28 15:33:00.874  4861  4861 F DEBUG   :     r8 bebe5074  r9 00000001  sl 90809528  fp 90809518
05-28 15:33:00.874  4861  4861 F DEBUG   :     ip bebe4d98  sp bebe4dc8  lr 8f9b19a8  pc 00000000  cpsr 60000010
05-28 15:33:00.966  4861  4861 F DEBUG   : 
05-28 15:33:00.966  4861  4861 F DEBUG   : backtrace:
05-28 15:33:00.966  4861  4861 F DEBUG   :     #00 pc 00000000  <unknown>
05-28 15:33:00.967  4861  4861 F DEBUG   :     #01 pc 0005a9a4  /data/app/com.companyname.xacrashapp-1/lib/arm/libaot-Mono.Android.dll.so (Android_Runtime_AndroidTypeManager_RegisterNativeMembers_Java_Interop_JniType_System_Type_string+852)
05-28 15:33:00.967  4861  4861 F DEBUG   :     #02 pc 00067564  /data/app/com.companyname.xacrashapp-1/lib/arm/libaot-Mono.Android.dll.so (Android_Runtime_JNIEnv_RegisterJniNatives_intptr_int_intptr_intptr_int+600)
05-28 15:33:00.967  4861  4861 F DEBUG   :     #03 pc 0000e970  <anonymous:90c71000>

this same stack trace can be seen as part of https://github.com/xamarin/xamarin-android/issues/2966 , down to an identical native offset from that method. Investigating if it is a duplicate of this / if the debugging log there applies here.

This is also showing the same crash location as https://github.com/xamarin/xamarin-android/issues/2521

FYI: I was only able to reproduce using the non-accelerated ARM v7a Nougat VM.

[alexanderkyte]

Workaround:

Put

<AndroidAotAdditionalArguments>llvmopts="-data-layout="p:32:32:32" -O2 -disable-tail-calls"</AndroidAotAdditionalArguments>

into the .csproj .

My emulator is a bit slow to launch it, but it definitely doesn't crash, whereas previously it crashed quite often.

@brendanzagaeski it looks like the fix for that bug didn't make it to upstream / that the fix didn't fix it.

Looking at the IR files in question:

[aot-compiler stdout] Mono Ahead of Time compiler - compiling assembly /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/android/assets/Xamarin.Android.Support.Core.Utils.dll
    [aot-compiler stdout] AOTID 0BE43ECF-B2CF-5CBA-A955-E2FD932244A3
    [aot-compiler stdout] Executing opt: "/Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/Darwin/opt" -f -O2 -disable-tail-calls -o "/Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.opt.bc" "/Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.bc"
    [aot-compiler stdout] Executing llc: "/Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/Darwin/llc"  -mattr=+vfp2,-neon,+d16 -asm-verbose=false -mtriple=armv7-linux-gnueabi -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=mono_aot_Xamarin_Android_Support_Core_Utils_eh_frame -disable-tail-calls -relocation-model=pic -o "/Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp-llvm.s" "/Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.opt.bc"
    [aot-compiler stdout] Code: 0(0%) Info: 1284(22%) Ex Info: 497(8%) Unwind Info: 0(0%) Class Info: 643(11%) PLT: 74(0%) GOT Info: 2368(41%) Offsets: 911(15%) GOT: 392
    [aot-compiler stdout] Compiled: 154/154 (100%), LLVM: 154 (100%), No GOT slots: 13 (8%), Direct calls: 0 (100%)
    [aot-compiler stdout] Executing the native assembler: "/Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-as"   -mfpu=vfp3 -o /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.s.o /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.s
    [aot-compiler stdout] Executing the native assembler: "/Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-as"   -mfpu=vfp3 -o /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp-llvm.o /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp-llvm.s
    [aot-compiler stdout] Executing the native linker: "/Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ld"  -shared -o /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/libaot-Xamarin.Android.Support.Core.Utils.dll.so.tmp "/Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp-llvm.o" /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll/temp.s.o /Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/libgcc.a /Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/platforms/android-21/arch-arm/usr/lib/libc.so /Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/platforms/android-21/arch-arm/usr/lib/libm.so
    [aot-compiler stdout] Stripping the binary: "/Users/akyte/Library/Developer/Xamarin/android-ndk/android-ndk-r15c/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-strip" --strip-symbol=\$a --strip-symbol=\$d /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/libaot-Xamarin.Android.Support.Core.Utils.dll.so.tmp
    [aot-compiler stdout] JIT time: 126 ms, Generation time: 601 ms, Assembly+Link time: 78 ms.
    [aot-compiler stdout] Mono Ahead of Time compiler - compiling assembly /Users/akyte/Projects/XACrashApp/XACrashApp/obj/Release/android/assets/Xamarin.Android.Support.Vector.Drawable.dll
    [aot-compiler stdout] AOTID 9C0267A9-462D-1EBA-37B2-8FA7682DBBC4

Using methodology:

llvm-dis temp.bc
llvm-dis temp.opt.bc
llvm-diff temp.opt.ll temp.ll 2>&1

to produce the diff that opt makes, and we can then diff-the-diff to see how differences in passing the workaround flags changes code that opt produces.

With workaround:
https://gist.github.com/alexanderkyte/0062c6f84d255b7fe42def098b9df616

Without workaround:
https://gist.github.com/alexanderkyte/3857ea579220c82a9e87a74480b03217

Now the difference between the two is pretty stark:

This shows us that whatever we are embedding into the emitted assembly, it is not having the same impact on ARM v7a as simply passing those options to opt.

[vargaz]

Do the emitted files have the layout attribute ? Is it the same as the one passed on the opt command line ?

[alexanderkyte]

https://gist.github.com/alexanderkyte/5b665596823ebe81987ff053c8aaaa34

@vargaz no it looks like we don't have any layout attributes baked into the .bc file

[brendanzagaeski]

@alexanderkyte (cc @vargaz), what version of the Xamarin.Android SDK are you testing with?

When I was working on the release notes for the Xamarin.Android SDK version 9.3, my understanding was that the fix to emit the layout attribute was only included starting in Xamarin.Android SDK version 9.3 and higher. At the moment that version is not yet published to the Stable updater channel in Visual Studio for Mac. It is available in the Preview channel of Visual Studio for Mac and in Visual Studio 2019 version 16.1 on Windows.

[alexanderkyte]

@vargaz are we sure that the fix is in VS2019?

ls /Library/Frameworks/Mono.framework/Versions          
5.18.1  Current

akyte@Alexanders-MacBook-Pro-3 ~/Projects/XACrashApp/XACrashApp/obj/Release/aot/armeabi-v7a/Xamarin.Android.Support.Core.Utils.dll (master*) $ mono-sgen --version
Mono JIT compiler version 5.18.1.3 (2018-08/fdb26b0a445 Wed Mar 20 10:02:02 EDT 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS:           
    SIGSEGV:       altstack
    Notification:  kqueue
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug 
    Interpreter:   yes
    LLVM:          yes(600)
    Suspend:       preemptive
    GC:            sgen (concurrent by default)

or
https://github.com/mono/mono/tree/fdb26b0a4454f60d20df1ea6c01fd851ffa4084a
but
https://github.com/mono/mono/commits/2018-08

is way ahead of that.

Let's confirm that your fixes are in there.

[alexanderkyte]

@brendanzagaeski ah, I was reproducing with Stable, because it seemed that everyone else here was reporting issues around the Stable release (VS2019 (16.0.2) ).

[alexanderkyte]

@brendanzagaeski let me try to reproduce with XA from Preview

[tobiasschulz]

I'm having the issue with VS2019 on windows 16.1.1

[alexanderkyte]

VS2019 from Preview channel using:

Mono JIT compiler version 5.18.1.24 (2018-08/082e1a23463 Tue Apr 30 13:26:14 EDT 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS:           
    SIGSEGV:       altstack
    Notification:  kqueue
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug 
    Interpreter:   yes
    LLVM:          yes(600)
    Suspend:       preemptive
    GC:            sgen (concurrent by default)

Does not reproduce the deterministic crash with the consistent native offset, and the resulting app seems very stable.

[brendanzagaeski]

I'm having the issue with VS2019 on windows 16.1.1

I double-checked the behavior on Visual Studio 2019 version 16.1.1 on Windows and was able to successfully run the app built with AOT Compilation and LLVM enabled on an arm64-v8a device.

@tobiasschulz I suspect you might be seeing a different but similarly problematic crash https://github.com/xamarin/xamarin-android/issues/3112. That crash affects a wider number of scenarios because it does not depend on having LLVM or AOT Compilation enabled.

[EmilAlipiev]

my app is not crashing immediately but crashes on some certain pages and i dont see anything special about those pages. I try to debug in release mode but although i enabled "enable developer instrumentation", i am still not able to get any log. Crash log doesnt also appear in Appcenter. So how can i find the reason why it crashes? the same app with same settings works just fine using VS2017

[brendanzagaeski]

Thanks again for the report! I will close this as a duplicate of https://github.com/xamarin/xamarin-android/issues/2521 to tidy up the issue list. On Windows, the fix is currently available in Visual Studio 2019 version 16.1, but on Mac it's only in the Preview channel of Visual Studio for Mac at the moment. I will update the status of https://github.com/xamarin/xamarin-android/issues/2521 to Closed when the fix has been published to the Stable channel of Visual Studio for Mac.

[beavis88]

Still not working on VS2019 16.2.0.

[brendanzagaeski]

@beavis88 , if you could report a new issue with the full error message you are seeing (for example, including the output from adb logcat -d after the app crashes if you are seeing an app crash), that would be perfect. Thanks in advance!

[beavis88]

@brendanzagaeski , thank you for pointing out, issue created.