Godot: Android, GLES2, Rigged Meshes kill the app on Nexus 7 (2013)

[wombatstampede]

Godot version:
3.1 stable (same problem on 3.1.1RC1)
GLES2, Android

OS/device including version:
Nexus 7 (2013), Squalcomm Snapdragon S4 Pro, Adreno 320
Android 6.0 Marshmallow
Other devices where this occurs:
Nexus 7 (2012), Tegra 3
Asus ME302C (Intel Atom, PowerVR)

Issue description:
Whenever a rigged object "comes into view", the app just stops (logcat just notes a "WindowState: WIN DEATH") without any further errors. I tried it out with two different rigged objects. When tracing with GAPID the log stops directly after compiling the Fragment Shader for the Rigged Object (glCompileShader). I would expect a "glGetShaderiv" after this but this isn't issued anymore.

The mesh/app works on other Platforms (i.e. Android device with Adreno 418 or Windows Device).

Logcat output (doesn't help much IMHO):

04-22 13:15:18.233   599  1299 I ActivityManager: Start proc 28351:org.godotengine.shaderfail/u0a225 for activity org.godotengine.shaderfail/org.godotengine.godot.Godot
04-22 13:15:18.234 28351 28351 I art     : Late-enabling -Xcheck:jni
04-22 13:15:18.337  1174  1174 W Binder_9: type=1400 audit(0.0:2736): avc: denied { ioctl } for path="socket:[535794]" dev="sockfs" ino=535794 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.337  1174  1174 W Binder_9: type=1400 audit(0.0:2737): avc: denied { ioctl } for path="socket:[535794]" dev="sockfs" ino=535794 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.377   503   503 W Binder_4: type=1400 audit(0.0:2738): avc: denied { ioctl } for path="socket:[542145]" dev="sockfs" ino=542145 ioctlcmd=7704 scontext=u:r:surfaceflinger:s0 tcontext=u:r:surfaceflinger:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.377   503   503 W Binder_4: type=1400 audit(0.0:2739): avc: denied { ioctl } for path="socket:[542145]" dev="sockfs" ino=542145 ioctlcmd=7704 scontext=u:r:surfaceflinger:s0 tcontext=u:r:surfaceflinger:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.384 28351 28366 D OpenGLRenderer: Use EGL_SWAP_BEHAVIOR_PRESERVED: true
04-22 13:15:18.377   503   503 W Binder_4: type=1400 audit(0.0:2740): avc: denied { ioctl } for path="socket:[535796]" dev="sockfs" ino=535796 ioctlcmd=7704 scontext=u:r:surfaceflinger:s0 tcontext=u:r:surfaceflinger:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.377   503   503 W Binder_4: type=1400 audit(0.0:2741): avc: denied { ioctl } for path="socket:[535796]" dev="sockfs" ino=535796 ioctlcmd=7704 scontext=u:r:surfaceflinger:s0 tcontext=u:r:surfaceflinger:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.407  1024  1024 W Binder_4: type=1400 audit(0.0:2742): avc: denied { ioctl } for path="socket:[541030]" dev="sockfs" ino=541030 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.407  1024  1024 W Binder_4: type=1400 audit(0.0:2743): avc: denied { ioctl } for path="socket:[541030]" dev="sockfs" ino=541030 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.439 28351 28366 I Adreno-EGL: <qeglDrvAPI_eglInitialize:379>: QUALCOMM Build: 10/21/15, 369a2ea, I96aee987eb
04-22 13:15:18.448 28351 28366 I OpenGLRenderer: Initialized EGL, version 1.4
04-22 13:15:18.494 28351 28365 W GodotView: creating OpenGL ES 2.0 context :
04-22 13:15:18.554 28351 28365 I godot   : OpenGL ES 2.0 Renderer: Adreno (TM) 320
04-22 13:15:18.685   599   617 I ActivityManager: Displayed org.godotengine.shaderfail/org.godotengine.godot.Godot: +465ms
04-22 13:15:18.688 28351 28365 W AudioTrack: AUDIO_OUTPUT_FLAG_FAST denied by client; transfer 1, track 44100 Hz, output 48000 Hz
04-22 13:15:18.677   759   759 W Binder_3: type=1400 audit(0.0:2744): avc: denied { ioctl } for path="socket:[542159]" dev="sockfs" ino=542159 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.677   759   759 W Binder_3: type=1400 audit(0.0:2745): avc: denied { ioctl } for path="socket:[542159]" dev="sockfs" ino=542159 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.687  3261  3261 W Binder_D: type=1400 audit(0.0:2746): avc: denied { ioctl } for path="socket:[542158]" dev="sockfs" ino=542158 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.687  3261  3261 W Binder_D: type=1400 audit(0.0:2747): avc: denied { ioctl } for path="socket:[542158]" dev="sockfs" ino=542158 ioctlcmd=7704 scontext=u:r:system_server:s0 tcontext=u:r:system_server:s0 tclass=unix_stream_socket permissive=0
04-22 13:15:18.693 18526 18526 I GoogleInputMethod: onFinishInput() : Dummy InputConnection bound
04-22 13:15:18.694   195   634 D audio_hw_primary: out_set_parameters: enter: usecase(1: low-latency-playback) kvpairs: routing=2
04-22 13:15:18.694 18526 18526 I GoogleInputMethod: onStartInput() : Dummy InputConnection bound
04-22 13:15:18.705   195   634 D audio_hw_primary: select_devices: out_snd_device(2: speaker) in_snd_device(0: none)
04-22 13:15:18.705   195   634 D ACDB-LOADER: ACDB -> send_afe_cal
04-22 13:15:18.705   195   634 D audio_hw_primary: enable_snd_device: snd_device(2: speaker)
04-22 13:15:18.715   195   634 D audio_hw_primary: enable_audio_route: apply and update mixer path: low-latency-playback
04-22 13:15:18.930 16055 16095 I Icing   : Indexing com.google.android.gms-apps from com.google.android.gms
04-22 13:15:19.096 16055 16095 I Icing   : Indexing com.google.android.gms-internal.3p:MobileApplication from com.google.android.gms
04-22 13:15:19.123 16055 16095 I Icing   : Indexing done com.google.android.gms-apps
04-22 13:15:19.129 16055 16095 I Icing   : Indexing done com.google.android.gms-internal.3p:MobileApplication
04-22 13:15:19.135 18526 18526 I IcingNotification: Received intent: Intent { act=com.google.android.gms.icing.IME_NOTIFICATION flg=0x10 pkg=com.google.android.inputmethod.latin (has extras) }
04-22 13:15:19.140 18526 18526 I IcingNotification: Received intent: Intent { act=com.google.android.gms.icing.IME_NOTIFICATION flg=0x10 pkg=com.google.android.inputmethod.latin (has extras) }
04-22 13:15:21.537   195   634 D AudioFlinger: mixer(0xb4140000) throttle end: throttle time(11)
04-22 13:15:22.530   599  1150 D GraphicsStats: Buffer count: 5
04-22 13:15:22.530   599  1025 I WindowState: WIN DEATH: Window{53071a7 u0 org.godotengine.shaderfail/org.godotengine.godot.Godot}
04-22 13:15:22.530   599  1025 W WindowManager: Force-removing child win Window{20bc03e u0 SurfaceView} from container Window{53071a7 u0 org.godotengine.shaderfail/org.godotengine.godot.Godot}
04-22 13:15:22.536   599   647 E qcom_sensors_hal: hal_process_report_resp: Result: 1, Error: 5
04-22 13:15:22.536   599   647 E qcom_sensors_hal: hal_process_report_resp: 6 Error: 3 Reason: 0
04-22 13:15:22.538   599   647 E qcom_sensors_hal: hal_process_report_resp: Result: 1, Error: 5
04-22 13:15:22.538   599   647 E qcom_sensors_hal: hal_process_report_resp: 0 Error: 4 Reason: 16
04-22 13:15:22.538   599   610 E qcom_sensors_hal: hal_smgr_report_delete: Error in report delete
04-22 13:15:22.539   599  1150 W WindowManager: Failed looking up window
04-22 13:15:22.539   599  1150 W WindowManager: java.lang.IllegalArgumentException: Requested window android.os.BinderProxy@2c729fd does not exist
04-22 13:15:22.539   599  1150 W WindowManager:     at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:8736)
04-22 13:15:22.539   599  1150 W WindowManager:     at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:8727)
04-22 13:15:22.539   599  1150 W WindowManager:     at com.android.server.wm.WindowState$DeathRecipient.binderDied(WindowState.java:1209)
04-22 13:15:22.539   599  1150 W WindowManager:     at android.os.BinderProxy.sendDeathNotice(Binder.java:558)
04-22 13:15:22.539   599  1150 I WindowState: WIN DEATH: null
04-22 13:15:22.540 28032 28396 I AKMD2   : AKMD_Release: Memory free.
04-22 13:15:22.590   205   205 I Zygote  : Process 28351 exited cleanly (1)
04-22 13:15:22.629   599  3261 I ActivityManager: Process org.godotengine.shaderfail (pid 28351) has died
04-22 13:15:22.631   599  3261 W ActivityManager: Force removing ActivityRecord{ba50ed5 u0 org.godotengine.shaderfail/org.godotengine.godot.Godot t2937}: app died, no saved state

Steps to reproduce:
Add a rigged mesh to your scene.
Run it on a Nexus 7 (2013).
The app should directly stop when attempting to show that scene (or if that object comes into view -> shader compiles).

Minimal reproduction project:
shader_fail.zip

(Edit: Just tried it with the rigged player object of the 3d platformer demo -> same effect)

[lawnjelly]

I have had I believe the same problem on Nexus 7 2012 (Nvidia Tegra 3) with skinned objects. I get crash running your shader_fail.

It may be related to this:
https://gamedev.stackexchange.com/questions/77854/how-can-i-reliably-implement-gpu-skinning-in-android

04-22 14:02:07.510 3267-3280/org.godotengine.shaderfail W/GodotView: creating OpenGL ES 2.0 context :
04-22 14:02:07.540 3267-3280/org.godotengine.shaderfail I/godot: OpenGL ES 2.0 Renderer: NVIDIA Tegra 3
04-22 14:02:07.640 482-498/? I/ActivityManager: Displayed org.godotengine.shaderfail/org.godotengine.godot.Godot: +1s124ms
04-22 14:02:10.500 3267-3267/org.godotengine.shaderfail D/dalvikvm: GC_FOR_ALLOC freed 374K, 5% free 7782K/8192K, paused 23ms, total 23ms
04-22 14:02:13.730 3267-3280/org.godotengine.shaderfail A/libc: Fatal signal 11 (SIGSEGV) at 0x00000010 (code=1), thread 3280 (Thread-147)
04-22 14:02:13.830 121-121/? I/DEBUG: * *
04-22 14:02:13.830 121-121/? I/DEBUG: Build fingerprint: 'google/nakasi/grouper:4.4.4/KTU84P/1227136:user/release-keys'
04-22 14:02:13.830 121-121/? I/DEBUG: Revision: '0'
04-22 14:02:13.830 121-121/? I/DEBUG: pid: 3267, tid: 3280, name: Thread-147 >>> org.godotengine.shaderfail <<<
04-22 14:02:13.830 121-121/? I/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000010
04-22 14:02:14.040 121-121/? I/DEBUG: r0 00001401 r1 00001401 r2 00000000 r3 6d606078
04-22 14:02:14.040 121-121/? I/DEBUG: r4 687b7d64 r5 00000010 r6 00000014 r7 6a61b018
04-22 14:02:14.050 121-121/? I/DEBUG: r8 6a3dee98 r9 6d605190 sl 00000000 fp 6d6052f8
04-22 14:02:14.050 121-121/? I/DEBUG: ip 00000001 sp 6d604d80 lr 6b9c6c04 pc 6b9c6ea4 cpsr 60010010
04-22 14:02:14.050 121-121/? I/DEBUG: d0 3f8000003f800000 d1 000000003f800000
04-22 14:02:14.050 121-121/? I/DEBUG: d2 00000000c1aed8d1 d3 0000000000000000
04-22 14:02:14.050 121-121/? I/DEBUG: d4 4090000080000000 d5 3f80000041aed8d1
04-22 14:02:14.050 121-121/? I/DEBUG: d6 3f8000003f800000 d7 3f8000003f800000
04-22 14:02:14.050 121-121/? I/DEBUG: d8 bf8000003f4ccccd d9 c1ac1d6e3c23d70a
04-22 14:02:14.050 121-121/? I/DEBUG: d10 3f00000041aed8d1 d11 0000000000000000
04-22 14:02:14.050 121-121/? I/DEBUG: d12 3f8000003f4ccccd d13 c1994f133c23d70a
04-22 14:02:14.050 121-121/? I/DEBUG: d14 3f5674043f800000 d15 c18c595e3f0bce72
04-22 14:02:14.050 121-121/? I/DEBUG: d16 0000000000000000 d17 00000000c17f9d7a
04-22 14:02:14.050 121-121/? I/DEBUG: d18 8000000080000000 d19 c1aed8d180000000
04-22 14:02:14.050 121-121/? I/DEBUG: d20 00000000bd803156 d21 bd80315680000000
04-22 14:02:14.050 121-121/? I/DEBUG: d22 0000000000000000 d23 00000000bd803156
04-22 14:02:14.050 121-121/? I/DEBUG: d24 bf29d08aa236578d d25 3fe54f19e4ebec8c
04-22 14:02:14.050 121-121/? I/DEBUG: d26 3fd7e1cb1c913900 d27 bf568a3524854b49
04-22 14:02:14.050 121-121/? I/DEBUG: d28 3fea367e59fe7b37 d29 bfe7519248885a30
04-22 14:02:14.050 121-121/? I/DEBUG: d30 3fc1d2cd2e5baeae d31 3feaf26460000000
04-22 14:02:14.050 121-121/? I/DEBUG: scr 60000093
04-22 14:02:14.050 121-121/? I/DEBUG: backtrace:
04-22 14:02:14.050 121-121/? I/DEBUG: #00 pc 007e6ea4 /data/app-lib/org.godotengine.shaderfail-1/libgodot_android.so
04-22 14:02:14.050 121-121/? I/DEBUG: #01 pc 007e6c00 /data/app-lib/org.godotengine.shaderfail-1/libgodot_android.so
04-22 14:02:14.050 121-121/? I/DEBUG: stack:
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d40 00001950
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d44 6d502668
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d48 00006540
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d4c 68ad0e58
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d50 6d6052f8
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d54 6b280568 /data/app-lib/org.godotengine.shaderfail-1/libgodot_android.so
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d58 00000018
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d5c 687b7d64
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d60 00000010
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d64 00000014
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d68 0000001c
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d6c 6a3dee98
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d70 6d605190
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d74 68ad0e58
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d78 6d6052f8
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d7c 6b9c6c04 /data/app-lib/org.godotengine.shaderfail-1/libgodot_android.so
04-22 14:02:14.050 121-121/? I/DEBUG: #00 6d604d80 0000001c
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d84 00000014
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d88 00000000
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d8c 2ec00000
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d90 00000020
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d94 00000004
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d98 401ab1d0
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604d9c 00000004
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604da0 6a387c08
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604da4 4016df23 /system/lib/libc.so (dlmalloc+4254)
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604da8 6d6050f0
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604dac 6d605100
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604db0 6d605110
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604db4 6d605260
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604db8 6d605270
04-22 14:02:14.050 121-121/? I/DEBUG: 6d604dbc 6d605280
04-22 14:02:14.050 121-121/? I/DEBUG: memory near r0:
04-22 14:02:14.050 121-121/? I/DEBUG: 000013e0 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.050 121-121/? I/DEBUG: 000013f0 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.050 121-121/? I/DEBUG: 00001400 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001410 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001420 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001430 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001440 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001450 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001460 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001470 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001480 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 00001490 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 000014a0 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 000014b0 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 000014c0 ffffffff ffffffff ffffffff ffffffff
04-22 14:02:14.060 121-121/? I/DEBUG: 000014d0 ffffffff ffffffff ffffffff ffffffff

I ended up implementing a fallback software skinning in a non-godot game. Is there a software skinning path available in godot?

[wombatstampede]

I am unsure if the option "USE_SKELETON_SOFTWARE" might be that. This depends on the presence of the GLES extension GL_ARB_texture_float or GL_OES_texture_float. I'll try to build an own export template forcing the use of USE_SKELETON_SOFTWARE. (Might take a while)

[lawnjelly]

Having a look at the shader code for skinning, unless USE_SKELETON_SOFTWARE is defined it looks like it is trying to use vertex texture fetch. This is not mandated I believe in GLES2.

The GLcaps page for Nexus 7 2012 shows GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS is 0, so afaik it can't do vertex texture fetch:
https://opengles.gpuinfo.org/displayreport.php?id=674

Doing a find in files for GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS doesn't report any hits, so I'm not sure Godot is even doing a test for vertex texture fetch before trying to use the technique, which would be a bug.

I've managed to get the android exporter compiling but not had any joy so far with USE_SKELETON_SOFTWARE, but I have no idea where to force it as yet.

I'm not actually sure that USE_SKELETON_SOFTWARE is actually doing software skinning, maybe just an alternative hardware method (not had enough time to look in detail). I would have thought a software skinning fallback to seem a good idea, I am surprised if it does not exist, but perhaps it requires a change to the architecture for skinned meshes..

Will look again at this when I have more time, interested in any more input from others, I've had limited time to debug so far.

[wombatstampede]

I just looked it up and the Nexus 7 2012 and 2013 use completely different chipsets and capabilities. The 2012 edition uses a Tegra 3 Chipset/GPU and the 2013 edition uses a Adreno 320 GPU (in a Qualcomm Snapdragon Chipset).

According to the website you posted, the Tegra 3 Chipset offers neither GL_ARB_texture_float nor GL_OES_texture_float. So it should #define USE_SKELETON_SOFTWARE in its skeleton shaders (according to the godot gles2 source). Therefore it doesn't use the "pose texture" which seems intended behaviour.

The Adreno 320 GPU however states to support GL_OES_texture_float so it doesn't define USE_SKELETON_SOFTWARE in godot (and also has a value of 16 for GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS).

It seems both GPUs/Apps stall for different reasons on the same rigged mesh. I personally built an own android template which forced USE_SKELETON_SOFTWARE on (via the related config variable in the GLES2 shader). Anyway, the app still failed, just a bit earlier (in the example around Shader compile 20 whereas the unchanged template failed around shader compile 48). So it didn't even reach a the "skeleton shader". Perhaps I did something wrong building the template, I don't know.

For my app, I'll for now simply try to hide all rigged meshes when I suspect a compatibility issue. But I know that there are many, many apps where rigged meshes are an essential part. It would be very helpful if all these can work at least in GLES2.

[nazgulsenpai]

Just for reference, tested your project on my Motorola G4 Play and it runs without issue.

Qualcomm MSM8916 Snapdragon 410
Adreno 306
Android 7.1.2

Edit: Tested on 3.1 stable and 3.2 Nightly Build (https://hugo.pro/projects/godot-builds/)

[wombatstampede]

Thank you. So it is not a general Adreno 3xx issue. I just encountered the issue again on an Android 4.2 Intel Atom (ME302C with PowerVR SGX 544MP2) tablet.

[lawnjelly]

I've now spent a bit of time trying to track down the crash on my Nexus 7 2012. I don't know whether I have a very efficient way of doing this but I have been repeatedly compiling the android export template and running the android version from the Godot IDE, and placing print_line statements throughout the rendering to find the offending code.

Mine is using the USE_SOFTWARE_SKELETON path, and the crash is on this line,
line 1456 of file rasterizer_scene_gles2.cpp:

PoolVector<uint8_t>::Read vertex_array_read = s->data.read();
const uint8_t *vertex_data = vertex_array_read.ptr();

vertex_data is NULL causing a crash, and the s->data.size() is 0.

I will try and track down this further, I don't know who wrote the original code, maybe they would have some ideas. I'm using wombatstampede's simple scene with a cat rigged mesh.

[lawnjelly]

I've pinned this down as the source of the crash on mine. I can get the rigged cat to display and work on the Android device by hard coding the bone IDs and weights and not trying to load from vertex_data if it is not present, as follows:

// do magic
size_t bones[4];
float bone_weight[4];

// fix ... hard coded values for bone IDs and weights
bones[0] = 0;
bones[1] = 0;
bones[2] = 0;
bones[3] = 0;
bone_weight[0] = 1.0f;
bone_weight[1] = 0.0f;
bone_weight[2] = 0.0f;
bone_weight[3] = 0.0f;

// on the Android device vertex_data is NULL, on desktop it is valid
if (vertex_data)
{
    if (s->attribs[VS::ARRAY_BONES].type == GL_UNSIGNED_BYTE) {
        // read as byte
        const uint8_t *bones_ptr = vertex_data + bones_offset + (i * bones_stride);
        bones[0] = bones_ptr[0];
        bones[1] = bones_ptr[1];
        bones[2] = bones_ptr[2];
        bones[3] = bones_ptr[3];
    } else {
        // read as short
        const uint16_t *bones_ptr = (const uint16_t *)(vertex_data + bones_offset + (i * bones_stride));
        bones[0] = bones_ptr[0];
        bones[1] = bones_ptr[1];
        bones[2] = bones_ptr[2];
        bones[3] = bones_ptr[3];
    }

    if (s->attribs[VS::ARRAY_WEIGHTS].type == GL_FLOAT) {
        // read as float
        const float *weight_ptr = (const float *)(vertex_data + bone_weight_offset + (i * bone_weight_stride));
        bone_weight[0] = weight_ptr[0];
        bone_weight[1] = weight_ptr[1];
        bone_weight[2] = weight_ptr[2];
        bone_weight[3] = weight_ptr[3];
    } else {
        // read as half
        const uint16_t *weight_ptr = (const uint16_t *)(vertex_data + bone_weight_offset + (i * bone_weight_stride));
        bone_weight[0] = (weight_ptr[0] / (float)0xFFFF);
        bone_weight[1] = (weight_ptr[1] / (float)0xFFFF);
        bone_weight[2] = (weight_ptr[2] / (float)0xFFFF);
        bone_weight[3] = (weight_ptr[3] / (float)0xFFFF);
    }
} // if vertex data

Being unfamiliar with the code I don't yet know where it loads the data for:
RasterizerStorageGLES2::Surface *s = static_cast<RasterizerStorageGLES2::Surface *>(p_element->geometry);
This may actually be easy to solve as it sounds like it may be a CPU side problem rather than GPU, perhaps there is an incorrect #ifdef or check for hardware caps that is prevent this data being loaded correctly in the Android build.

_Edit_
I am now getting the impression that the problem is that the bone IDs and weights are being stored on the GPU (with the idea that they might be needed there for the other skinning method?).
PoolVector<uint8_t> RasterizerStorageGLES2::mesh_surface_get_array(RID p_mesh, int p_surface)
contains this:

#ifndef TOOLS_ENABLED
ERR_PRINT("OpenGL ES 2.0 does not allow retrieving mesh array data");
#endif
return surface->data;

So perhaps this is the root of the problem, it is trying to read these back each frame and failing. Begs the question of whether the USE_SOFTWARE_SKELETON pathway works at all on GLES2 devices, can anyone confirm this? (maybe it has been copied from GLES3?)

If this is the problem then a more generic solution for this and similar problems of reading back from the vertex buffer might be to optionally create a software copy when updating it, and read from this copy in main memory.

[akien-mga]

For the reference, seems related to this older issue: #26972. But this issue has a lot more useful debugging information so it should be kept as the main one.

BTW, as per #27232, it seems that it worked fine in earlier 3.1 alpha builds.

[akien-mga]

CC @clayjohn in case you want to poke at this one.

[lawnjelly]

I now have a possible fix, I am still testing it (and crossing fingers), it is of course possible it might break something else(!).

In rasterizer_storage_gles2.cpp, line 2414, in function mesh_add_surface:

#ifdef TOOLS_ENABLED
    surface->blend_shape_data = p_blend_shapes;
    if (surface->blend_shape_data.size()) {
        ERR_PRINT_ONCE("Blend shapes are not supported in OpenGL ES 2.0");
    }
    surface->data = array;
    surface->index_data = p_index_array;
#endif

This is not adding the bone data to the surface when not in tools (desktop). If instead we change it to this:

#ifdef TOOLS_ENABLED
    surface->blend_shape_data = p_blend_shapes;
    if (surface->blend_shape_data.size()) {
        ERR_PRINT_ONCE("Blend shapes are not supported in OpenGL ES 2.0");
    }
    surface->index_data = p_index_array;
#endif
    surface->data = array;

It seems to fix the bug, with the cat at least I think for me. I'll try with some more complex skinned meshes. I would also question whether the index_data may also benefit from being set outside of tools as well, but I don't know, it doesn't seem necessary for the software skinning.

Hopefully someone who knows this code can see if this is a sensible fix, or better placed elsewhere. I also don't know github yet so if it does prove to be a workable solution I would appreciate if someone else could change in git.

It also may be possible to save a bit of memory by only adding this for skinned meshes for instance.

_Edit_
Now tested with an animated model other than wombat's cat, and it works!! Yay!! :+1:
https://www.youtube.com/watch?v=PbPBuo2d58E

[lawnjelly]

Slight improvement to the fix, old:

#ifdef TOOLS_ENABLED
    surface->blend_shape_data = p_blend_shapes;
    if (surface->blend_shape_data.size()) {
        ERR_PRINT_ONCE("Blend shapes are not supported in OpenGL ES 2.0");
    }
    surface->data = array;
    surface->index_data = p_index_array;
#endif

fixed:

#ifdef TOOLS_ENABLED
    surface->blend_shape_data = p_blend_shapes;
    if (surface->blend_shape_data.size()) {
        ERR_PRINT_ONCE("Blend shapes are not supported in OpenGL ES 2.0");
    }
    surface->data = array;
    surface->index_data = p_index_array;
#else

    // Fix for skinning bug with USE_SKELETON_SOFTWARE path.
    // On tablets etc (i.e. where TOOLS_ENABLED is false) there was no copy of
    // the bones and bone weights made into main memory on the surface. This is
    // assumed to be present later in the skinning code and would cause a crash if this data
    // is not copied here.
    // NOTES
    // This data is only needed specifically for the skinning, in other cases it could be a waste of memory.
    // However there may be other bugs in the future that could be solved by
    // similar method of retaining a copy in some cases,
    // where readback from the vertex data would be useful.
    // An alternative to hard coding here could be to pass into this function a bool
    // indicating whether a copy in main memory should be made.

    // If this geometry is used specifically for skinning
    if (p_format & (VS::ARRAY_FORMAT_BONES | VS::ARRAY_FORMAT_WEIGHTS))
    {
        // If USE_SKELETON_SOFTWARE is active (note this should flag should be tested
        // directly rather than indirectly, this can lead to bugs)
        if (!config.float_texture_supported)
            // Make a main memory copy of the vertex data
            surface->data = array;
    }
#endif

On wombatstampede's suggestion I've also created a fixed APK if anyone wants to test this on devices which currently crash due to the bug:

https://github.com/lawnjelly/GodotTweaks/blob/master/SkinningFix/GodotSkinning_LawnjellyFix1.apk

_Note this same fix may also fix crashes on iOS if there are devices that use GLES 2 but do not support float texture._

[wombatstampede]

The error still occurs on my Nexus 7 2013 = Adreno 320 (which uses another GPU than the Nexus 7 2012 => Tegra 3).

I even tried to force USE_SKELETON_SOFTWARE by setting config.float_texture_supported to false (in rasterizer_storage_gles2.cpp). I tried this on the master branch and checked that #29751 was merged.
(The device supports float textures so USE_SKELETON_SOFTWARE normally isn't defined.)

The result is the same: As soon as I enable the visibility of a rigged mesh (tried an own mesh (cat) and the character of the 3d platformer demo) the app stops after compiling (glCompileShader) of the fragment shader but before calling glGetShaderiv. The only logcat msg I get is "WindowState: WIN DEATH: Window{74c3ef4 u0 org.godotengine.shaderfail/org.godotengine.godot.Godot}".
GAPID Trace displays that the Fragment shader is compiled though.

Anyway, perhaps this only affects a small number of devices.

[lawnjelly]

I apologize wombat as there do seem to be 2 separate issues causing crashes with the rigged meshes, and mine has taken over this thread. I'd be quite keen to pin down yours too but it is difficult without some hardware that replicates it, maybe we can work through fixing your bug together? Definitely it was interesting that you had a crash with my test APK as I believe I had the USE_SKELETON_SOFTWARE hard coded, I would want to double check this.

I don't know how easy the issues are to edit but would it be acceptable to create a new issue for this so as to avoid confusing the two? My fault entirely. :smile:

[wombatstampede]

Absolutely not your fault. :-)

I am slightly clueless about this issue. But there is nearly no code between glCompileShader and glGetShaderiv. So my guess that this is some GPU issue which is triggered by the shader. So it would probably mean testing the shader separately and if that kills the app as well then removing code bit by bit until it runs again.

I have the shader source code from GAPID but I am unsure how to separately implement/compile that shader inside godot (i.e. to change it step by step). Especially concerning the uniforms+values (as far as that is relevant).

[wombatstampede]

Thanks for reopening. In a small debugging session (assisted by lawnjelly) I narrowed down the error to the inside of glLinkProgram so the error seems to occur in the Adreno 320 GPU.

I then changed the skeleton specific part in scene.glsl.

This works!

//  if (skeleton_in_world_coords) {
        bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
        world_matrix = bone_transform * world_matrix;
//  } else {
//      world_matrix = world_matrix * bone_transform;
//  }

This also works!

//  if (skeleton_in_world_coords) {
//      bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
//      world_matrix = bone_transform * world_matrix;
//  } else {
        world_matrix = world_matrix * bone_transform;
//  }

This code (current code) stalls the app:

    if (skeleton_in_world_coords) {
        bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
        world_matrix = bone_transform * world_matrix;
    } else {
        world_matrix = world_matrix * bone_transform;
    }

But this works as well: (same with if (1==0) {)

//  if (skeleton_in_world_coords) {
    if (1==1) {
        bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
        world_matrix = bone_transform * world_matrix;
    } else {
        world_matrix = world_matrix * bone_transform;
    }

At the moment I see nothing wrong with the uniform skeleton_in_world_coords. This bool is set in rasterizer_scene_gles2.cpp from skeleton->use_world_transform.
Maybe some different length/alignment for bools? Other bools seem to work therefore I'm slightly clueless.

[clayjohn]

I wonder if it is an issue of exceeding a maximum instruction count. That would explain why commenting out either branch works.

Unfortunately, with device specific bugs like this it is impossible for others to provide much assistance without being able to reproduce.

Keep up the good work!

[lawnjelly]

Brilliant, well done for pinning this down wombat! :smile:

As well as the max instruction count that clayjohn mentions, it reminds me that I have encountered problems before with GPUs on GLES2 before that refused to run shaders with branches. It is notable that this is branching on a uniform though which shouldn't be as big a deal as branching on an attribute.

[wombatstampede]

Still no real success. I did some testing with the Adreno Profiler and it only fails if the if() statement gets something to work. Bad to test because this gives no error but just the GLES driver simply fails (on the device, not in godot).

Is it possible to replace the uniform _skeleton_in_world_coords_ with a _#define SKELETON_IN_WORLD_COORDS_ ?

    #ifdef SKELETON_IN_WORLD_COORDS
        bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
        world_matrix = bone_transform * world_matrix;
    #else
        world_matrix = world_matrix * bone_transform;
    #endif

An #ifdef won't kill the app (tried it out). It might also slightly (very slightly) improve the performance. It won't work correctly if the same shader is actually reused for two armatures which have _bones_in_world_transform_ set to different values. (unsure if that ever happens)

[thimenesup]

Modified the 3.1.1 release branch and implemented #29751 and can confirm that in my device with ARM Mali-400 MP2 will still crash whenever a skinned mesh runs an animation

[lawnjelly]

Modified the 3.1.1 release branch and implemented #29751 and can confirm that in my device with ARM Mali-400 MP2 will still crash whenever a skinned mesh runs an animation

We did find there were at least 2 possible points of failure in the skinning code, the #29751 fix (Nexus 7 2012), and also the problem on wombatstampede's Nexus 7 2013. There may be more. Maybe you can try his fix above in scene.glsl and let us know if this cures the problem?

//  if (skeleton_in_world_coords) {
//      bone_transform = skeleton_transform * (bone_transform * skeleton_transform_inverse);
//      world_matrix = bone_transform * world_matrix;
//  } else {
        world_matrix = world_matrix * bone_transform;
//  }

[thimenesup]

Alright so I tested @wombatstampede changes in addition to the fix and they exhibit the same behaviour as the release branch, fine until the skinned mesh runs an animation, on the next frame the bones are completly glitched/broken and the engine crashes.

[thimenesup]

I tried running it again a couple times and in one of them it actually played the animation about 20 frames before crashing, with glitched/broken bones too the entire animation (That said the rotation
of the bones were correct? It seemed like it was similar to the animation it was supposed to play). Attached the errors the Android Monitor logged.
log.txt

[lawnjelly]

Just to confirm that you built the android templates (i.e. ran scons with platform android) and linked these in the export in your export settings .. well worth testing this by putting in some print_line somewhere in the source code and ensuring it comes up on your adb logcat (I've made this mistake of not actually using the built template myself many times!).

Assuming the export template is being built and used okay then it sounds like a new issue, it would be better to create new issue and fill out the template so we can investigate with a minimum reproduction project, perhaps a godot project and also an APK that exhibits the problem. It does sound like a different issue to me as both the earlier problems would crash before drawing the skinned mesh.

[thimenesup]

Alright so I found the culprit of the crash, atleast on my device. It runs the hardware skeleton branch

for (int i = 0; i < 4; i++) {
    ivec2 tex_ofs = ivec2(int(bone_ids[i]) * 3, 0);

    highp mat4 b = mat4(
        texel2DFetch(bone_transforms, skeleton_texture_size, tex_ofs + ivec2(0, 0)),
        texel2DFetch(bone_transforms, skeleton_texture_size, tex_ofs + ivec2(1, 0)),
        texel2DFetch(bone_transforms, skeleton_texture_size, tex_ofs + ivec2(2, 0)),
        vec4(0.0, 0.0, 0.0, 1.0));

    bone_transform += transpose(b) * bone_weights[i];
}

and changing the texel fetchs to raw vec4s makes it run without crashing (But the mesh won't be visible obviously)

[lawnjelly]

Ah.. I think I picked that up as a possible issue before. Was wondering if your hardware supported vertex texture fetch:

http://opengles.gpuinfo.org/displayreport.php?id=2233
GL_MAX_VERTEX_TEXTURE_IMAGE_UNITS 0

This suggests not. I had previously done a search in the source and couldn't find it, which suggests this cap is not being checked in Godot. If there is no vertex texture fetch we should be switching to the SKELETON SOFTWARE path. Should be relatively easy to fix, I can have a look (or anyone else who is familiar with this bit).

If you want to just bodge it for now, if you find the bit that tests the hardware for config.float_texture_supported:

rasterizer_storeage_gles2.cpp, line 5609:
config.float_texture_supported = config.extensions.has("GL_ARB_texture_float") || config.extensions.has("GL_OES_texture_float");
and hard code it to false, hopefully that will force the skeleton software path.

I should be able to do a PR to fix this properly in the next day or 2, it only requires 3 or so lines changing.

[thimenesup]

Confirmed that the issue is detecting the hardware support, forcing it to run the software one runs the animation with no issues.

[earlroxas13]

Hi guys,
just reporting that I still experience this bug on my Asus Zenfone 5 (2014), this issue only appear on this particular device, I tested on my older tablet with 512MB RAM, and this issue not appear.
The engine I use is 3.2 b 4
This is the logcat:
benchmark_3d_logcat.txt
Hope it helps. Thanks.

[lawnjelly]

Hi guys,
just reporting that I still experience this bug on my Asus Zenfone 5 (2014), this issue only appear on this particular device, I tested on my older tablet with 512MB RAM, and this issue not appear.
The engine I use is 3.2 b 4
This is the logcat:
benchmark_3d_logcat.txt
Hope it helps. Thanks.

This really needs its own issue I think. There are several different bugs that can affect skinning, it is better to have one issue per bug, and this appears to be a separate bug. In addition are you sure this is related to skinning (from looking at the log) or is it just a crash on startup?

[earlroxas13]

Yes, the game runs ok when opening scene with only 2D nodes, and or MeshInstance without animation and skeleton (tested with only cube primitive mesh). The game forced close only when opening scene with animated mesh.
Also, for information. As I said before this issue only appear on my Zenfone5 (2014), I tested on my Tablet with 512MB RAM (2015), the game runs normally, just slow because low spec.

[akien-mga]

@earlroxas13 Please open a new issue, with details about the hardware and a project that reproduces the issue.

[earlroxas13]

Oh, I thought that the issue is similar. so i just comment here.
Sorry, I'll open new issue then.

[akien-mga]

There are several issues discussed here, most of which I believe are now fixed in 3.2.

If anyone can still reproduce a related issue, please open a new issue (possibly linking back to this one for reference as there's a lot of relevant debug info here, but not much structure).