Godot: arm64 libgodot_android.so crash Android 10

[JezSonic]

3.2.1.stable

ANDROID 10/Google Pixel 3 XL:

Issue description:
Google Play Console found my game crash on Android 10/Google Pixel 3 Xl. Here is the backtrace.
I don't know what is the source of that issue cause it didn't happen on any of my devices. Anyway i am creating an issue here.

Google Pixel 3 XL (crosshatch), 3584MB RAM, Android 10
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.karolstudios.pixelz <<<

backtrace:
  #00  pc 0000000001087904  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #01  pc 00000000006cfdec  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #02  pc 000000000101eb1c  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #03  pc 000000000109dc94  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #04  pc 0000000000307840  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #05  pc 00000000002e64f8  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #06  pc 00000000006bd6dc  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #07  pc 000000000101d040  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #08  pc 00000000006be924  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #09  pc 00000000006c2790  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #10  pc 00000000006b384c  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #11  pc 000000000101eb1c  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #12  pc 0000000001018014  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #13  pc 0000000001018260  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #14  pc 00000000006db35c  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #15  pc 000000000017bc60  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so
  #16  pc 00000000001536d8  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/lib/arm64/libgodot_android.so (Java_org_godotengine_godot_GodotLib_step+164)
  #17  pc 00000000000101b0  /data/app/com.karolstudios.pixelz-oAB0kwpMxkRmsYnb5k0XKw==/oat/arm64/base.odex (art_jni_trampoline+144)
  #18  pc 0000000002004344  /memfd:/jit-cache (org.godotengine.godot.GodotRenderer.onDrawFrame+84)
  #19  pc 00000000020107a8  /memfd:/jit-cache (android.opengl.GLSurfaceView$GLThread.guardedRun+3976)
  #20  pc 000000000013663c  /apex/com.android.runtime/lib64/libart.so (art_quick_osr_stub+60)
  #21  pc 0000000000333acc  /apex/com.android.runtime/lib64/libart.so (art::jit::Jit::MaybeDoOnStackReplacement(art::Thread*, art::ArtMethod*, unsigned int, int, art::JValue*)+1660)
  #22  pc 00000000005a30f0  /apex/com.android.runtime/lib64/libart.so (MterpMaybeDoOnStackReplacement+212)
  #23  pc 0000000000135350  /apex/com.android.runtime/lib64/libart.so (MterpHelpers+240)
  #24  pc 00000000002d2e3e  /system/framework/framework.jar (android.opengl.GLSurfaceView$GLThread.guardedRun+682)
  #25  pc 000000000059a8a4  /apex/com.android.runtime/lib64/libart.so (MterpInvokeDirect+1168)
  #26  pc 0000000000130914  /apex/com.android.runtime/lib64/libart.so (mterp_op_invoke_direct+20)
  #27  pc 00000000002d35cc  /system/framework/framework.jar (android.opengl.GLSurfaceView$GLThread.run+48)
  #28  pc 00000000002b03a8  /apex/com.android.runtime/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.15430740532710954497)+240)
  #29  pc 000000000058980c  /apex/com.android.runtime/lib64/libart.so (artQuickToInterpreterBridge+1012)
  #30  pc 000000000013f468  /apex/com.android.runtime/lib64/libart.so (art_quick_to_interpreter_bridge+88)
  #31  pc 0000000000136334  /apex/com.android.runtime/lib64/libart.so (art_quick_invoke_stub+548)
  #32  pc 000000000014506c  /apex/com.android.runtime/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+244)
  #33  pc 00000000004a9b0c  /apex/com.android.runtime/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
  #34  pc 00000000004aaba0  /apex/com.android.runtime/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue const*)+416)
  #35  pc 00000000004ea93c  /apex/com.android.runtime/lib64/libart.so (art::Thread::CreateCallback(void*)+1176)
  #36  pc 00000000000e1100  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36)
  #37  pc 0000000000083ab0  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)

[Calinou]

I'm afraid a backtrace without debugging symbols isn't of much use, especially if the user who got the crash isn't around to reply to our questions.

[mrimvo]

I get quite a lot of remote crash reports (Crashlyics) from my android game, that look exactly like this (crash in libgodot_android.so), and to a lesser extend crashes in other .so binaries too. Please, how can I attach debug symbols to these reports? I would like to help you guys fix these problems.

In the crashlytics docu they say I need to execute
./gradlew crashlyticsUploadSymbolsRelease
but in a godot project I have no idea where/how to execute that?
Please, could you guide us in a general direction?

This problem is rather urgend, because crashes like this produce 1-star ratings in google play for games that would have great ratings otherwise. Quite a big bummer!

Most crash reports I get look like this:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.mvolution.cuarentena1 <<<

backtrace:
  #00  pc 00000000006fe9bc  /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
  #01  pc 000000000127327c  /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
  #02  pc 0000000000076278  /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
  #03  pc 0000000000009137  /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/oat/arm/base.odex (offset 0x9000)

Some other .so libraries report crashes too, but not as many.

My users complain about the app crashing on startup - others about a black screen, sound is starting but nothing more.

I have Crashlytics up and running and would like to upload debugging symbols to the Crashlytics backend to provide you guys with meaningful backtraces from various devices out there in the wild. I'm kind of lost on how to do that. Any ideas?

[Calinou]

cc @m4gr3d

[jeremyz]

@mrimvo
you might have to recompile godot with modified ./platform/android/java/lib/build.gradle and ./platform/android/java/app/build.gradle ?
My game suffers the same, I want it to be fixed ASAP.
How did you setup crashlytics, is it just a matter of registering your project ?

[mrimvo]

Yes, it's quite easy to set up, but as the crash reports are not meaningful, this is a bit useless. Recompiling might help I didn't try that. Another thing that just crossed my mind: maybe it's possible to add godot's logs (where all the push_error messages end up) to a crash report. Because I think some of these errors might be rooted id GDScript. The engine shouldn't crash, but at least we can do something about it if we had the logs.

Edit: I did a quick research and I think the idea to attach the godot logs might not be possible with Crashlytics:

the CrashlyticsListener doesn't support being called from when an NDK/JNI exception happens

Another option might by to use Crashlytics.log as soon as there are messages added to godot's terminal output. However i found no way how to get notified about terminal output in godot.

[akien-mga]

To have a full debug stacktrace when it crashes, you should build custom export templates with target=debug instead of target=release_debug (what "Debug" templates use by default).

They will be significantly bigger but will include full debug symbols so that the crash in libgodot_android.so can give us a detailed stacktrace.

---

Also to be sure, please verify that you're are building the export templates properly from the same commit and local modifications as the editor. If you're using the "Custom Build" option, you should also reinstall the custom compiled android_source.zip in place of your pre-existing res://android/build/ which might have been installed with the official 3.2.1-stable (which is thus not compatible with your custom changes).

---

Edit: I might have misinterpreted that you have custom changes made to the template. If you're using the stock Android template from the Godot 3.2.1 templates package, then you just need to rebuild a custom debug templates from the 3.2.1-stable git tag, with:

scons p=android tools=no target=debug android_arch=arm64v8
scons p=android tools=no target=debug android_arch=armv7
cd platform/android/java
./gradlew generateGodotTemplates

And use the generated bin/android_debug.apk as debug template here:

And export with "Debug" on of course so that this debug template is used.

[mrimvo]

Thank you for your answer :)

I'm using a custom android build template. I'm happy to push a debug build into the beta channel of my app to help godot get some helpful stack traces.

I'm a bit confused about where to add target=debug to build a debug template from my custom template. Could you elaborate on that?

Just to be clear about my context:

• Using Godot 3.2.1 stable
• Used the menu: Project > Install Android Build Template
• Changed that template to my needs (eg added Crashlytics)
• Godot uses that template for my android exports

[akien-mga]

See the last part of my previous comment, you'd need to build the Godot export template with debug symbols (see Compiling for Android).

Then it's a bit tricky as the new workflow with custom build templates doesn't allow to easily use one's own android_source.zip to set things up (until #36728 is fixed).

Since you're using 3.2.1-stable as is, the best might be that once you have built the export template using the above instructions with target=debug, you can copy bin/godot-lib.debug.aar in place of your installed res://android/build/libs/debug/godot-lib.debug.aar. I haven't tested but that should add the necessary debug symbols while still allowing you to use the template source installed from 3.2.1-stable.

[mrimvo]

I tried to follow that path, but I'm sad to say I need to give up on that. The NDK Download is just too big for my limited data volume to handle.

[akien-mga]

I'll make a debug build myself and upload it here, that should be much easier to test.

[akien-mga]

Here's a target=debug build of 3.2.1-stable for Android (arm64v8 and armv7): https://send.firefox.com/download/ce4ac3341d7a2601/#XRIXZIyiWUh2oZkBpYSqAg (available for 7 days)

How to use it:

• If you don't use "Custom Template/Use Custom Build", then select the android_debug.apk as "Custom Template/Debug" in your Android preset, and export a Debug build
• If you use "Custom Template/Use Custom Build", either:
  
  
  
  • Delete res://android, put the downloaded android_source.zip in your Godot templates/3.2.1.stable folder, use "Install Android Build Template..." from the editor and redo your local changes, or
  
  
  • Keep your res://android, unzip android_source.zip somewhat and copy its android_source/libs/debug/godot-lib.debug.aar to res://android/build/libs/debug/ to replace the existing one. (Recommended as you don't need to mess with your installed Godot templates and redo your Android customization, but I'm only 95% confident that doing this is sufficient - should be good enough though :))
  
  

[mrimvo]

This is awesome! I'm going to upload the debug version today! :+1:

If you don't use "Custom Template/Use Custom Build", then select the android_debug.apk as "Custom Template/Debug" in your Android preset, and export a Debug build

This worked for me to build and run the game with your template apk.

If you use "Custom Template/Use Custom Build", either:

I tried both ways, and even did not re-apply my changes, so I had the pure version as Custom Template set. It produces errors while building a debug version of the game:

I then removed the references to these resources from the AndroidManifest.xml and got another error while building:

Also, I noticed the godot-lib.debug.aar is

• the only notable change compared to what I had before in the android/ folder
• it's size is only ~20MB while the original was about 40 MB

[jeremyz]

size is smaller because it contains arm64v8 and armv7, no x86 and x86_64.
@mrimvo maybe run ./gradlew clean from within ./android/build ?

[jeremyz]

I can build and run using _custom_template/debug_ and the given android_debug.apk
or using _custom_template/use_custom_build_ and replacing godot-lib.debug.aar with the one from the given android_source.zip

to enable crashlytics I edit _build.graddle_ :
+ classpath 'com.google.gms:google-services:4.3.3'
+ classpath 'com.google.firebase:firebase-crashlytics-gradle:2.1.0'
+ apply plugin: 'com.google.firebase.crashlytics'
+ implementation 'com.google.firebase:firebase-crashlytics:17.0.0'

but I must comment out implementation libraries.supportCoreUtils
because of a Manifest merger failed between com.android.support:support-compat:28.0.0 from AndroidManifest.xml and androidx.core:core:1.0.0 androidx.core.app.CoreComponentFactory.
I'm not sure if this will work, it seams like.

How can I publish one of these debug apk, google's policy does not allow debuggable applications ?

[mrimvo]

I just got notified by google that my app got suspended. Oh man that hurts. Their reasoning being it got covid-19 references. Well it does but I think their reaction is a bit harsh, provided it was nothing more than a cute little game.

So I'm out of this. About publishing the godot-lib.debug.aar I think it might be possible to mix and have the exported APK be a release version that uses the debug version of godot-lib. Another (better) possiblity would be to use godot-lib.release.aar and to upload the debug information to Crashlytics backend.

About enabling Crashlytics: yes, that's exactly the way I did it too, including to remove the support-compat. Also you need to download and add google-services.json from Crashlytics backend and add it to android/build/ directory.

[jeremyz]

Hoooo sorry, that's harsh !
I already tried to publish a release apk built with the android_debug.apk, it has be detected and refused. I'll complete crashlytics setup and upload the debug symbols to crashlytics.
Yes, I forgot to mention google-services.json ...

[jeremyz]

That was hard !
Crashlytics does not work for me ...
*.so libs are stripped in the apk, but I finally learned about ndk-stack tool.
give him a dir where unstripped libraries are and you're good.
So has I thought, problems comes from JSON serialisation ...

GDScript code is : fout.store_line(JSON.print(dict))

bt is :

godot-3.2.1-stable/core/object.cpp:941:33
godot-3.2.1-stable/core/variant.cpp:1592:28
godot-3.2.1-stable/core/variant.cpp:1408:9
godot-3.2.1-stable/core/io/json.cpp:117:33
godot-3.2.1-stable/core/io/json.cpp:111:10
godot-3.2.1-stable/core/io/json.cpp:111:10
godot-3.2.1-stable/core/io/json.cpp:111:10
godot-3.2.1-stable/core/io/json.cpp:123:9
godot-3.2.1-stable/core/bind/core_bind.cpp:3210:9
godot-3.2.1-stable/./core/method_bind.gen.inc:2505:17
godot-3.2.1-stable/core/object.cpp:921:17
`
full_bt

[akien-mga]

Here's the start of @jeremyz's backtrace in the 3.2.1-stable codebase:

https://github.com/godotengine/godot/blob/3.2.1-stable/core/object.cpp#L941

https://github.com/godotengine/godot/blob/3.2.1-stable/core/variant.cpp#L1592

https://github.com/godotengine/godot/blob/3.2.1-stable/core/variant.cpp#L1408

https://github.com/godotengine/godot/blob/3.2.1-stable/core/io/json.cpp#L117

https://github.com/godotengine/godot/blob/3.2.1-stable/core/io/json.cpp#L111

So it sounds like the dict you're serializing containers Objects with scripts (I assume Nodes?), but then trying to call to_string() from their script crashes (probably null pointer dereference?).

That sounds like it might be related to #38119 (CC @RandomShaper), which is fixed in 3.2.2 beta 2 and beta 3. Would you be able to test if 3.2.2 beta 3 fixes the issue for you?

[RandomShaper]

Yes, sounds a lot like the object, along with its script, has been freed.

Since the dangling Variant fix doesn't apply to release builds, it won't help you unless you find what's wrong in your scripts.

Running in the editor with the debugger attached, you'll get those freed objects converted to the "[Deleted Object]" string.

As a quick measure to avoid the crash and maybe gain some time for finding the bug before you get more crash reports, you can use a release_debug export template. With it, you won't get a crash neither the "[Deleted Object]" string, but "[Object:null]".

[jeremyz]

I've noticed an object that should not be here in the dict that I'm serializing.
reading your comments I'm now almost sure that the crash happens when I save the turn state having just destroyed that object ... will look deeper into it in a few hours ...

@RandomShaper I haven't play with the debugger yet, cause it only crashed on android 10 so far ... will have a deeper look it this too.
@akien-mga I'll test 3.2.2 behaviour ... (can you please triage my 2 EventGesture OS agnostic PR, so that I know where it's going ;)

[mrimvo]

Well done @jeremyz I'm glad you got the backtrace working. 🙂

Besides fixing the game itself, I would like to suggest that godot should not crash in these cases but instead pushes errors like push_error does.

[RandomShaper]

There's a plan about having the option to have those checks in release builds too and even add further protections.

I haven't had time to write the proposal yet.

[jeremyz]

I've tested the current situation,
jasonify a dictionary with a dangling reference.

var d = {'stuff':'None'}
var a = Node2D.new()
d['dangling'] = a
print(JSON.print(d))
a.queue_free()
yield(get_tree().create_timer(1.0), "timeout")
print(JSON.print(d))

with 3.2.1-stable

• release -> segfault
• release_debug -> [Object:0] object.cpp:946
• debugger -> [Deleted Object] variant.cpp 1588

with 3.2 HEAD(6abe73f5b8)

• release -> segfault
• release_debug -> [Object:null] from variant.cpp:1608
• debugger -> [Deleted Object] variant.cpp 1605

@akien-mga so, as stated in #38119's updated comment, crashes are not handled in release builds yet.