Runtime: Android: PAL_VirtualUnwind failed

[qmfrederik]

Running a .NET Core application on Android, with CoreCLR and CoreFX cross-built for arm64 Android, results in this error message:

LD_LIBRARY_PATH=/data/local/tmp/coredroid/ /data/local/tmp/coredroid/corerun /data/local/tmp/coredroid/helloworld.dll

Assert failure(PID 1306 [0x0000051a], Thread: 1306 [0x051a]): !"Thread::VirtualUnwindToFirstManagedCallFrame: PAL_VirtualUnwind failed"
    File: /home/fcarlier/git/coreclr/src/vm/stackwalk.cpp Line: 770
    Image: /data/local/tmp/coredroid/corerun

Aborted

[qmfrederik]

@janvorli I've created a new issue since we managed to solve dotnet/runtime#7507 . I'll try to get some more information and post an update here.

@cydhaselton Are you seeing the same issue?

[qmfrederik]

So it turns out that unw_init_local returns -8. According to libunwind-common.h, that's -UNW_EINVAL, which is described as:

unw_init_local() was called in a version of libunwind which supports remote unwinding only (this normally happens when calling unw_init_local() for a cross-platform version of libunwind).

[qmfrederik]

@janvorli I got a bit confused by this piece of the documentation at http://manpages.ubuntu.com/manpages/zesty/man3/libunwind.3.html

The principle behind supporting native, cross-platform, and
multi-platform unwinding is very simple: for native unwinding,
program includes and uses the linker switch -lunwind.
For cross-platform unwinding, a program includes and
uses the linker switch -lunwind-PLAT, where PLAT is the name of the
target platform (e.g., ia64 for IA-64, hppa-elf for ELF-based HP
PA-RISC, or x86 for 80386). Multi-platform unwinding works exactly like
cross-platform unwinding, the only limitation is that a single source
file (compilation unit) can include at most one libunwind header file.
In other words, the platform-specific support for each supported target
needs to be isolated in separate source files---a limitation that
shouldn't be an issue in practice.

This seems to indicate you only need to include libunwind.h and link with unwind, and not include libunwind-aarch64.h and link with libunwind-aarch64

Do you think it makes sense to try __not__ linking with libunwind-aarch64? (I'll probably give it a try anyway but it would be good to get your take on this one)

[cydhaselton]

@qmfrederik Yes, I'm seeing the same issue. For the record, I've always run into issues when attempting to just include libunwind.h and link to unwind; I attempted to do so after reading that same documentation.

The issues, if I remember correctly, were missing symbols that could be found when linking with libunwind-aarch64.

[janvorli]

@qmfrederik we are including only the libunwind.h, not the arch specific ones (well, we include libunwind-ptrace.h if it exists but that's a different thing).
As for linking to the libraries, your info is interesting and makes sense, but I believe we were hitting some undefined symbols when we didn't include the arch specific lib. Maybe it is just a bad factoring of the aarch64 stuff or something. It would be great if you could try to exclude the arch specific one and see what happens.

[cydhaselton]

@qmfrederik and @janvorli Re-built with unwind and ran into undefined symbols as before:

/build/core/coreclr/src/pal/src/exception/seh-unwind.cpp:180: undefined reference to_Uaarch64_get_reg'`

For some reason building libunwind for aarch64…even on aarch64…doesn't add the necessary symbols to libunwind

[janvorli]

@qmfrederik could you try to link just against unwind on your Linux ARM64 and see if it works there? If it does, it would mean there is some problem in the Android build of the libunwind.

[cydhaselton]

If it doesn't, there's a mirror of the llvm libunwind sources (https://github.com/llvm-mirror/libunwind) that has been patched to include a switch to build a native-only libunwind. I'm wondering if this would resolve the issue if used instead of Google's libunwind sources

Reference: http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160523/159802.html

[janvorli]

We are already using a modified version of the LLVM libunwind for CoreRT and I would eventually want to migrate coreclr to using it as well. But it will take a little time.

[qmfrederik]

The gist of the issues seems to be an error in build-android-rootfs:136, which triggers this condition in configure.ac:196 in the libunwind makefile.

Because libunwind is compiled with host x86_64 and target aarch64, libunwind is compiled with remote unwind support only and that triggers the runtime error we're seeing.

[janvorli]

@qmfrederik Ah, that makes sense then. And that's why it wanted the unwind-aarch64 during the build.

[janvorli]

@qmfrederik so can we just set both the host and target in the configure parameters to the same value or would it break the compilation itself?

[qmfrederik]

@janvorli That will be the fix, but it causes a lot of extra code to be compiled and introduces some new linking issues. Nothing which cannot be fixed; I'll continue working on it.

[cydhaselton]

@qmfrederik and @janvorli Ideally --build=build_arch, --host=arm_arch and --target=arm_arch...but that causes errors in the libunwind build for some reason.

[qmfrederik]

dotnet/coreclr#9940 should fix this - @cydhaselton feel free to give it a try, I now get the following program output:

fcarlier@ubuntu:~$ adb shell LD_LIBRARY_PATH=/data/local/tmp/coredroid/ /data/local/tmp/coredroid/corerun /data/local/tmp/coredroid/helloworld.dll

Unhandled Exception: System.TypeLoadException
   at System.Console.WriteLine(String value)
   at ConsoleApp.Program.Main(String[] args)
Aborted

[janvorli]

@qmfrederik That's great!

[qmfrederik]

@janvorli What would be the easiest way to get an idea about what causes the TypeLoadException? The lldb plugin doesn't work (yet) on Android so we can't step through managed code.

[qmfrederik]

... @janvorli setting a breakpoint at clrex.cpp:1588 gives this:

* thread dotnet/coreclr#1, name = 'corerun', stop reason = breakpoint 1.1
    frame #0: libcoreclr.so`EETypeLoadException::EETypeLoadException(this=0x0000005570df81e0, pszNameSpace=0x0000007fb688d5c9, pTypeName=0x0000007fb688f651, pAssemblyName=u"System.IO, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a", pMessageArg=0x0000000000000000, resIDWhy=2148734242) at clrex.cpp:1603
   1600     }
   1601     CONTRACTL_END;
   1602
-> 1603     if(pszNameSpace)
   1604     {
   1605         SString sNameSpace(SString::Utf8, pszNameSpace);
   1606         SString sTypeName(SString::Utf8, pTypeName);

let me check where that comes from

[qmfrederik]

and this is the call stack:

* thread dotnet/coreclr#1, name = 'corerun', stop reason = breakpoint 1.1
  * frame #0: libcoreclr.so`EETypeLoadException::EETypeLoadException(this=0x0000005586c036c0, pszNameSpace=0x0000007fb688d5c9, pTypeName=0x0000007fb688f651, pAssemblyName=u"System.IO, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a", pMessageArg=0x0000000000000000, resIDWhy=2148734242) at clrex.cpp:1588
    frame dotnet/coreclr#1: libcoreclr.so`ThrowTypeLoadException(pszNameSpace=0x0000007fb688d5c9, pTypeName=0x0000007fb688f651, pAssemblyName=u"System.IO, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a", pMessageArg=0x0000000000000000, resIDWhy=2148734242) at excep.cpp:13519
    frame dotnet/coreclr#2: libcoreclr.so`Assembly::ThrowTypeLoadException(this=0x0000005586c07440, pszNameSpace=0x0000007fb688d5c9, pszTypeName=0x0000007fb688f651, pszMethodName=0x0000000000000000, resIDWhy=2148734242) at assembly.cpp:2700
    frame dotnet/coreclr#3: libcoreclr.so`Assembly::ThrowTypeLoadException(this=0x0000005586c07440, pName=0x0000007fffff85c8, resIDWhy=2148734242) at assembly.cpp:2625
    frame dotnet/coreclr#4: libcoreclr.so`ClassLoader::LoadTypeHandleThrowIfFailed(this=0x0000005586c089d0, pName=0x0000007fffff85c8, level=CLASS_LOADED, pLookInThisModuleOnly=0x0000007f3d4fb178) at clsload.cpp:580
    frame dotnet/coreclr#5: libcoreclr.so`ClassLoader::LoadTypeDefOrRefThrowing(pModule=0x0000007f3d4f8400, typeDefOrRef=16777247, fNotFoundAction=ThrowIfNotFound, fUninstantiated=FailIfUninstDefOrRef, tokenNotToLoad=0, level=CLASS_LOADED) at clsload.cpp:3062
    frame dotnet/coreclr#6: libcoreclr.so`SigPointer::GetTypeHandleThrowing(this=0x0000007fffff8b38, pModule=0x0000007f3d4f8400, pTypeContext=0x0000007fffff8b48, fLoadTypes=LoadTypes, level=CLASS_LOADED, dropGenericArgumentLevel=NO, pSubst=0x0000000000000000, pZapSigContext=0x0000000000000000) const at siginfo.cpp:1493
    frame dotnet/coreclr#7: libcoreclr.so`CEEInfo::ConvToJitSig(pSig="", cbSig=4, scopeHnd=0x0000007f3d4f8400, token=0, sigRet=0x0000007fffffbe48, pContextMD=0x0000007f3d4f9988, localSig=false, contextType=TypeHandle @ 0x0000007fffff8b30) at jitinterface.cpp:550
    frame dotnet/coreclr#8: libcoreclr.so`CEEInfo::getMethodSigInternal(this=0x0000007fffffd3f8, ftnHnd=0x0000007f3d4f9988, sigRet=0x0000007fffffbe48, owner=0x0000007f3d4faf20) at jitinterface.cpp:8486
    frame dotnet/coreclr#9: libcoreclr.so`CEEInfo::getCallInfo(this=0x0000007fffffd3f8, pResolvedToken=0x0000007fffffbd50, pConstrainedResolvedToken=0x0000000000000000, callerHandle=0x0000007f3d4fa9c8, flags=17, pResult=0x0000007fffffbe38) at jitinterface.cpp:5711
    frame dotnet/coreclr#10: libclrjit.so`Compiler::eeGetCallInfo(this=0x0000007fb690b030, pResolvedToken=0x0000007fffffbd50, pConstrainedToken=0x0000000000000000, flags=17, pResult=0x0000007fffffbe38) at ee_il_dll.hpp:44
    frame dotnet/coreclr#11: libclrjit.so`Compiler::impImportBlockCode(this=0x0000007fb690b030, block=0x0000007fb69103d8) at importer.cpp:12765
    frame dotnet/coreclr#12: libclrjit.so`Compiler::impImportBlock(this=0x0000007fffffc1f8, pParam=0x0000007fffffc208)::$_1::operator()(Compiler::impImportBlock(BasicBlock*)::FilterVerificationExceptionsParam*) const at importer.cpp:15750
    frame dotnet/coreclr#13: libclrjit.so`Compiler::impImportBlock(this=0x0000007fb690b030, block=0x0000007fb69103d8) at importer.cpp:15760
    frame dotnet/coreclr#14: libclrjit.so`Compiler::impImport(this=0x0000007fb690b030, method=0x0000007fb69103d8) at importer.cpp:16834
    frame dotnet/coreclr#15: libclrjit.so`Compiler::fgImport(this=0x0000007fb690b030) at flowgraph.cpp:6672
    frame dotnet/coreclr#16: libclrjit.so`Compiler::compCompile(this=0x0000007fb690b030, methodCodePtr=0x0000007fffffcc78, methodCodeSize=0x0000007fffffd1fc, compileFlags=0x0000007fffffcc90) at compiler.cpp:4207
    frame dotnet/coreclr#17: libclrjit.so`Compiler::compCompileHelper(this=0x0000007fb690b030, classPtr=0x0000007f3d4f8400, compHnd=0x0000007fffffd3f8, methodInfo=0x0000007fffffd298, methodCodePtr=0x0000007fffffcc78, methodCodeSize=0x0000007fffffd1fc, compileFlags=0x0000007fffffcc90, instVerInfo=INSTVER_GENERIC_PASSED_VERIFICATION) at compiler.cpp:5804
    frame dotnet/coreclr#18: libclrjit.so`Compiler::compCompile(this=0x0000007fffffc7f8, __JITpParam=0x0000007fffffc800)::$_0::operator()(Compiler::compCompile(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*)::__JITParam*) const at compiler.cpp:5156
    frame dotnet/coreclr#19: libclrjit.so`Compiler::compCompile(this=0x0000007fb690b030, methodHnd=0x0000007f3d4fa9c8, classPtr=0x0000007f3d4f8400, compHnd=0x0000007fffffd3f8, methodInfo=0x0000007fffffd298, methodCodePtr=0x0000007fffffcc78, methodCodeSize=0x0000007fffffd1fc, compileFlags=0x0000007fffffcc90) at compiler.cpp:5176

Makes me none the wiser, all pointers appreciated :)

[janvorli]

@qmfrederik I would recommend starting from the other end - verifying that you have collected the assemblies the right way. Just copying output from the build of corefx and coreclr doesn't work for me and results in similar issue that you were hitting. The only reliable way I have found is to take advantage of the tests/runtests.sh script in the coreclr to collect the assemblies.
Here is what I do:
On my Windows machine, I build the tests using build.cmd with no parameters
Then on my Ubuntu 14.04:

• copy the files from my windows machine from bin\tests\Windows_NT.x64.Debug under the coreclr repo folder (exclude the bin and TestWrappers folders) to a local folder on Linux. Say it is ~/tests
• go to corefx repo folder
• ./src/Native/build-native.sh cross arm64
  *./build-managed.cs -buildArch=arm64
• go to coreclr repo folder
• ./build.sh arm64 cross skipnuget
  *./tests/runtest.sh --testRootDir=/home/janvorli/tests/ --testNativeBinDir=/home/janvorli/git/coreclr/bin/obj/Linux.arm64.Debug/tests --coreClrBinDir=/home/janvorli/git/coreclr/bin/Product/Linux.arm64.Debug --mscorlibDir=/home/janvorli/git/coreclr/bin/Product/Linux.arm64.Debug --coreFxBinDir=/home/janvorli/git/corefx/bin/runtime/netcoreapp-Linux-Debug-arm64
• Interrupt it after it starts running tests and failing
• Go to ~/tests/Tests/coreoverlay
• check that the native .so files from the corefx build are there (I think they should be, but I am not sure, maybe you'll need to copy them in manually)
• Now the coreoverlay folder should contain assemblies that match each other and so you can transfer it all to the android device and try it.

[janvorli]

@qmfrederik I can share the result of the tests build from Windows with you if you don't have Windows machine available / set up for build.

[qmfrederik]

@janvorli Thanks, I'll give it a try soon and let you know how it goes.

[cydhaselton]

@janvorli: Can the same procedure be used on Ubuntu 16.04...i.e. is it necessary to use a Windows machine as an intermediary?

[janvorli]

@cydhaselton unfortunately, there is no way to build the tests on Unix yet. It is one of the pain points that we would like to fix soon. There is a github issue dotnet/runtime#7217 for that.

[qmfrederik]

@cydhaselton As I understand it, one of the main issues is type forwarding between assemblies like System.IO.dll and System.Private.CoreLib.dll, and you need to make sure all the assemblies are in sync.

What @janvorli suggests is the best way to do so, as you're building from coreclr/master and corefx/master.

That said, another possibility is to use the lastest NuGet packages from MyGet. They include the latest versions of most dlls and should reasonably be in sync with System.Private.CoreLib you're cross building.

For example, I've just updated my helloworld.csproj file to use today's NuGet packages from CoreFX.
You should be able to do something like dotnet restore, dotnet publish (this will generate a x64 Linux build), and then update the .csproj file to remove the reference to the ubuntu.16.04-x64 runtime, do a dotnet restore and dotnet build (to get an AnyCPU version of the helloworld.dll assembly).
Then, take those files together with your CoreCLR + CoreFX native build and deploy them to Android (so what I talk about in the readme file.

And with that:

LD_LIBRARY_PATH=/data/local/tmp/coredroid/ ./corerun helloworld.dll                         <
Hello World!

Success, .NET Core's "Hello World" runs on Android!

[janvorli]

@qmfrederik that's really awesome!
CC: @gkhanna79, @Petermarcu

[qmfrederik]

Yup, pretty cool stuff! Thanks to @janvorli, @sdmaclea, @cydhaselton and the many others who helped to make this work :)

[cydhaselton]

Excellent!

@qmfrederik: So, can you post a list of files that need to be copied from Ubuntu to Android? Or do I need to run the build on Windows?

Also, are you running the dotnet restore dotnet publish on Windows or Ubuntu?

[gkhanna79]

This is super awesome! :)

[qmfrederik]

@gkhanna79 Next up: RIDs and CI builds for Android? ;-)

[cydhaselton]

My personal goal is to get powershell on android...

[qmfrederik]

@cydhaselton Doing everything from Ubuntu 16.04. My notes (although incomplete) are at https://github.com/qmfrederik/coredroid/blob/master/README.md; PRs are welcome ;-)

Here's a list of files I have on my Android device (probably I've copied over too many files, but it should give you an idea):

SOS.NETCore.dll
System.Collections.dll
System.Console.dll
System.Globalization.Native.so
System.Globalization.dll
System.IO.Compression.Native.so
System.IO.FileSystem.Primitives.dll
System.IO.dll
System.Native.so
System.Net.Http.Native.so
System.Net.Security.Native.so
System.Private.CoreLib.dll
System.Private.Uri.dll
System.Reflection.Primitives.dll
System.Reflection.dll
System.Resources.ResourceManager.dll
System.Runtime.Extensions.dll
System.Runtime.Handles.dll
System.Runtime.InteropServices.dll
System.Runtime.dll
System.Security.Cryptography.Native.OpenSsl.so
System.Security.Cryptography.Native.so
System.Text.Encoding.Extensions.dll
System.Text.Encoding.dll
System.Threading.Tasks.dll
System.Threading.dll
corerun
helloworld.dll
ilasm
ildasm
libandroid-glob.so
libandroid-support.so
libclrjit.so
libcoreclr.so
libdbgshim.so
libgnustl_shared.so
libicudata.so
libicudata.so.58
libicudata.so.58.2
libicui18n.so
libicui18n.so.58
libicui18n.so.58.2
libicuio.so
libicuio.so.58
libicuio.so.58.2
libicutest.so
libicutest.so.58
libicutest.so.58.2
libicutu.so
libicutu.so.58
libicutu.so.58.2
libicuuc.so
libicuuc.so.58
libicuuc.so.58.2
libintl.so
liblzma.so
libmscordaccore.so
libmscordbi.so
libsos.so
libsuperpmi-shim-collector.so
libsuperpmi-shim-counter.so
libsuperpmi-shim-simple.so
libuuid.so
libuuid.so.1
mcs
mscorlib.dll
superpmi

[gkhanna79]

@qmfrederik Definitely help bring them up and let us know how we can help. That said, it will be good to get the "How to build for Android" PRed into master.

[cydhaselton]

@qmfrederik Followed your instructions and got Unhandled Exception: System.BadImageFormatException when running helloworld.dll

-Is there anything you did in this PR that wasn't replicated to your instructions?-

Will add "update the .csproj file to remove the reference to the ubuntu.16.04-x64 runtime" to coredroid readme

[qmfrederik]

@cydhaselton I think the BadImageFormatException may be because your helloworld.dll targets amd64 instead of AnyCPU.

If you have access to a Windows machine, you can use ILSpy to check.

In short - in the helloworld project, remove the RuntimeIdentifier from the .csproj, run dotnet restore, dotnet build, and then copy the bin/Debug/netstandard2.0/helloworld.dll file over to your Android device (not the one from publish/ubuntu16.04-x64/). That should get you a helloworld.dll which targets AnyCPU.

[cydhaselton]

@qmfrederik Yup, already done & PR submitted. Will post other questions in Next Steps thread