Mixedrealitytoolkit-unity: Unity app hangs when MRTK enabled (arm64, il2cpp)

[darax]

Describe the bug

A clear and concise description of what the bug is.

To reproduce

Steps to reproduce the behavior:

1. Make an arm64 app for HL2 on unity 2019.1.14
2. enable the MRTK spatial mapping and input services
3. run the app for awhile
4. the rendering is blank

Expected behavior

Rendering continues

Screenshots

If applicable, add screenshots to help explain your problem.

Your Setup (please complete the following information)

• Unity Version [e.g. 2 2019.1.14]
• MRTK Version Microsoft Mixed Reality Toolkit 2.0.0

Target Platform (please complete the following information)

• HoloLens 2

Additional context

The Ordered systems list could be cached and only recreated when systems are added or removed.

This is going to generate a ton of garbage since it is called several times per frame:
private bool ExecuteOnAllServicesInOrder(Action execute)
{
var orderedSystems = MixedRealityServiceRegistry.GetAllServices().OrderBy(m => m.Priority);
return ExecuteOnAllServices(orderedSystems, execute);
}

CritSec GameAssembly!GC_allocate_ml+0 at 00007ffb29d2ccc0
WaiterWoken No
LockCount 2
RecursionCount 1
OwningThread 11a4
EntryCount 0
ContentionCount 7
* Locked

7 Id: 1020.11a4 Suspend: 1 Teb: 00000072`9a918000 Unfrozen

RetAddr : Args to Child : Call Site

00 00007ffb27460b7c : 000000729b4fe7c0 00007ffb27460b7c 00007ffb29a8a9e0 00007ffb29a8a9e0 : GameAssembly!GC_mark_from+0x13c [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\mark.c @ 839] 01 00007ffb27460858 : 000000729b4fe820 00007ffb27460858 0000000001345c3b 00007ffb29a8a9e0 : GameAssembly!GC_mark_some_inner+0x2ac [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPexternalbdwgcmark.c @ 389]
02 00007ffb27466940 : 000000729b4fe850 00007ffb27466940 00007ffb00000000 00007ffb29a8a9e0 : GameAssembly!GC_mark_some+0x10 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\mark.c @ 495] 03 00007ffb27466e58 : 000000729b4feb40 00007ffb27466e58 000000729b4fe860 00007ffb29a8a9e0 : GameAssembly!GC_stopped_mark+0x498 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPexternalbdwgcalloc.c @ 809]
04 00007ffb27459f7c : 000000729b4febb0 00007ffb27459f7c 0000000000001000 00007ffb29a8a9e0 : GameAssembly!GC_try_to_collect_inner+0x180 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\alloc.c @ 557] 05 00007ffb27459244 : 000000729b4fec10 00007ffb27459244 0000000000000000 0000000000000001 : GameAssembly!GC_collect_or_expand+0xc4 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPexternalbdwgcalloc.c @ 1461]
06 00007ffb2745d9d4 : 000000729b4fec90 00007ffb2745d9d4 0000000100000000 0000019000000000 : GameAssembly!GC_allocobj+0x294 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\alloc.c @ 1556] 07 00007ffb2745d6b0 : 000000729b4fecf0 00007ffb2745d6b0 0000000000000001 00007ffb29d2ccc0 : GameAssembly!GC_generic_malloc_inner+0x29c [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPexternalbdwgcmalloc.c @ 191]
08 00007ffb2745f9c0 : 000000729b4fed50 00007ffb2745f9c0 0000000000000000 0000000000000050 : GameAssembly!GC_generic_malloc+0x188 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\malloc.c @ 280] 09 (Inline Function) : ---------------- ---------------- ---------------- ---------------- : GameAssembly!GC_malloc_kind_global+0xc8 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\external\bdwgc\malloc.c @ 327] 0a 00007ffb27446ec0 : 000000729b4fed90 00007ffb27446ec0 0000000000000006 00000190b3d02430 : GameAssembly!GC_malloc_kind+0xe0 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPexternalbdwgcmalloc.c @ 333]
0b (Inline Function) : ---------------- ---------------- ---------------- ---------------- : GameAssembly!il2cpp::vm::Object::Allocate+0x4 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectIL2CPPlibil2cppvmObject.cpp @ 55]
0c 00007ffb2838be78 : 000000729b4fede0 00007ffb2838be78 000000729b4fede0 fffffffffffffffe : GameAssembly!il2cpp::vm::Array::NewSpecific+0xd8 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\libil2cpp\vm\Array.cpp @ 142] 0d 00007ffb280ff904 : 000000729b4fede0 00007ffb2838ba50 000000729b4fee70 00007ffb280ff904 : GameAssembly!Buffer_1__ctor_mFC593D644243CEC302C442E4AA893955A7EA307C_gshared+0x480 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectSourceil2cppOutputGenerics33.cpp @ 27418]
0e (Inline Function) : ---------------- ---------------- ---------------- ---------------- : GameAssembly!Buffer_1__ctor_mFC593D644243CEC302C442E4AA893955A7EA307C+0x1c [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectSourceil2cppOutputGenerics34.cpp @ 10512]
0f 00007ffb281e9190 : 000000729b4feec0 00007ffb281e9190 0000000000000000 0000000000000000 : GameAssembly!U3CGetEnumeratorU3Ed__1_MoveNext_m23D3773DBF4D81ABBC236E9DDA007553DC56A04F_gshared+0x54 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\Source\il2cppOutput\Generics34.cpp @ 14745] 10 (Inline Function) : ---------------- ---------------- ---------------- ---------------- : GameAssembly!InterfaceFuncInvoker0<bool>::Invoke+0x54 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\Source\il2cppOutput\Microsoft.MixedReality.Toolkit3.cpp @ 73] 11 00007ffb281e89f4 : 000000729b4feec0 00007ffb281e8ee4 000000729b4fefc0 00007ffb281e89f4 : GameAssembly!MixedRealityToolkit_ExecuteOnAllServices_m7F5DF4DD8D4C9FD2D42556B02325A47A94172FDF+0x360 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectSourceil2cppOutputMicrosoft.MixedReality.Toolkit3.cpp @ 12248]
12 00007ffb281f3ce8 : 000000729b4ff080 00007ffb281f3ce8 00000190ac3ed400 000000729b4ff010 : GameAssembly!MixedRealityToolkit_ExecuteOnAllServicesInOrder_m0562E6D760ADFEBE8D458F991EC49CF4E3BAD756+0x2c4 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\Source\il2cppOutput\Microsoft.MixedReality.Toolkit3.cpp @ 12017] 13 00007ffb281f3f70 : 000000729b4ff120 00007ffb281f3f70 00000190acb59630 000000729b4ff090 : GameAssembly!MixedRealityToolkit_UpdateAllServices_m1909290BC589EC6049EB06525D64083E6CDE0640+0x1f8 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectSourceil2cppOutputMicrosoft.MixedReality.Toolkit3.cpp @ 11336]
14 00007ffb271a84c4 : 000000729b4ff1a0 00007ffb271a84c4 00000190acb59630 000000729b4ff130 : GameAssembly!MixedRealityToolkit_Update_mEBE2065607C9518A5AA69B0BA56BC45AAC908144+0x1d0 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\Source\il2cppOutput\Microsoft.MixedReality.Toolkit3.cpp @ 9728] 15 00007ffb27442f14 : 000000729b4ff1b0 00007ffb27442f14 000000729b4ff220 00007ffb2b57ec1c : GameAssembly!RuntimeInvoker_TrueVoid_t22962CB4C05B1D89B55A6E1139F0E87A90987017+0x14 [E:gitsraasCapSRaaSvsbridgeIl2CppOutputProjectSourceil2cppOutputIl2CppInvokerTable.cpp @ 33964]
16 00007ffb2b57ec1c : 000000729b4ff220 00007ffb2b57ec1c 000000729b4ff310 fffffffffffffffe : GameAssembly!il2cpp::vm::Runtime::Invoke+0x94 [E:\git\sraas\CapSRaaS\vsbridge\Il2CppOutputProject\IL2CPP\libil2cpp\vm\Runtime.cpp @ 542] 17 00007ffb2b5945d0 : 000000729b4ff290 00007ffb2b5945d0 0000000000000000 fffffffffffffffe : UnityPlayer!scripting_method_invoke+0x6c
18 00007ffb2b598e3c : 000000729b4ff300 00007ffb2b598e3c 00000190acb59630 0000000000000000 : UnityPlayer!ScriptingInvocation::Invoke+0xa0 19 00007ffb2b598f28 : 000000729b4ff460 00007ffb2b598f28 0000000000000000 0000000000000000 : UnityPlayer!MonoBehaviour::CallMethodIfAvailable+0x9c
1a 00007ffb2afd4b34 : 000000729b4ff490 00007ffb2afd4b34 00000190acb59630 00000190acb59630 : UnityPlayer!MonoBehaviour::CallUpdateMethod+0xe0 1b 00007ffb2afdc69c : 000000729b4ff530 00007ffb2afdc69c fffffffffffffffe 000000729b4ff4a8 : UnityPlayer!BaseBehaviourManager::CommonUpdate+0x17c
1c 00007ffb2b284800 : 000000729b4ff570 00007ffb2b284800 fffffffffffffffe 00007ffb2d472a80 : UnityPlayer!BehaviourManager::Update+0x34 1d 00007ffb2b272584 : 000000729b4ff5a0 00007ffb2b272584 fffffffffffffffe 0000000000000008 : UnityPlayer!InitPlayerLoopCallbacks'::2'::UpdateScriptRunBehaviourUpdateRegistrator::Forward+0x48
1e 00007ffb2b272650 : 000000729b4ff780 00007ffb2b272650 00007ffb2c5c0058 0000000000000004 : UnityPlayer!ExecutePlayerLoop+0x5c 1f 00007ffb2b275c18 : 000000729b4ff960 00007ffb2b275c18 0000007200000001 00000190aad80020 : UnityPlayer!ExecutePlayerLoop+0x128
20 00007ffb2bcbaefc : 000000729b4ffa00 00007ffb2bcbaefc fffffffffffffffe 00007ffb2d483e18 : UnityPlayer!PlayerLoop+0x128 21 00007ffb2bca05f0 : 000000729b4ffa20 00007ffb2bca05f0 000000729b4ffa78 0000000000000000 : UnityPlayer!metro::MainLoop+0x24
22 00007ffb2bca7fa4 : 000000729b4ffac0 00007ffb2bca7fa4 0000000000000000 00007ffb2bca0a2c : UnityPlayer!UnityPlayer::AppCallbacks::DoPerformUpdateAndRender+0x1b8 23 00007ffb58dd97a8 : 000000729b4ffba0 00007ffb58dd97a8 0000000000000000 0000000000000000 : UnityPlayer!UnityPlayer::AppCallbacks::_AppThreadImplementation+0x304
24 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x58 [minkernelntdllrtlstrt.c @ 1152]

[wiwei]

@davidkline-ms, @julenka FYI

[wiwei]

Oddly enough, I was just looking at this due to some unrelated purposes and noticed this as well and was going to fix it. We never should be:

1) Re-allocating the same thing every single frame in a core update loop.
2) Be using Linq in a core update loop.

[wiwei]

https://github.com/microsoft/MixedRealityToolkit-Unity/pull/5948

[spatialsystemsdev]

@wiwei we are seeing this crash and using an older version of MRTK. other than avoiding more allocations, was there anything else done to solve this issue? any ideas why it was trigging a crash in the GC?

[darax]

https://forum.unity.com/threads/updating-boehm-il2cpp-garbage-collector.763244/ has a thread regarding the GC crash. I think this may be fixed in Unity 2020 for HoloLens. I've posted a workaround at the end of this message for you to try. I've also encountered this issue on iOS, but wasn't able to reproduce the issue outside of the project I was working on. I did something very lame-I disable GC during critical states that my users will lose work if there is a crash, and enable GC and force a collect when the project is doing something that won't cause data loss.

That being said, programmer error can also lead to the GC crash if something gets used after being freed or gets freed more than once. I know this first hand, having made such errors after identifying this bug. In this case, the frequent allocations make the problem more likely to surface, but do not directly cause the crash. As is often the case in memory corruption bugs, the corrupted stack is often an innocent bystander, and the offender is long gone.

I'm still on an older version of Unity with this bug, so I patch it at build time. You can try by making a script with this code and putting it in your project's assets in a folder named Editor.

public class BuildProcessing : IPostprocessBuildWithReport
{
    public int callbackOrder => 1;

    const string boehmGcPath = @"Il2CppOutputProject\IL2CPP\external\bdwgc\mark.c";
    public void OnPostprocessBuild(BuildReport report)
    {
        // Patch mark.c in the Boehm garbage collector and add wiFiControl capability.
        string basePath = report.summary.outputPath;

        // GC update 
        // from 
        // while ((((ptr_t)mark_stack_top - (ptr_t)mark_stack) | credit) >= 0)
        // to  
        //while (((((word)mark_stack_top - (word)mark_stack) | (word)credit)
        //&SIGNB) == 0)
        string markPath = Path.Combine(basePath, boehmGcPath);
        if (File.Exists(markPath))
        {
            string markText = File.ReadAllText(markPath);
            markText = markText.Replace("while ((((ptr_t)mark_stack_top - (ptr_t)mark_stack) | credit) >= 0)", "while (((((word)mark_stack_top - (word)mark_stack) | (word)credit)\r\n&SIGNB) == 0)");
            File.WriteAllText(markPath, markText);
        }
        else
        {
            Debug.Log("didn't find boehm mark.c");
        }
    }
}