Mixedrealitytoolkit-unity: The MRTK does not support being unloaded and destroyed

[provencher]

Describe the bug

Currently, if I use SceneManager.LoadScene to change scenes, the MRTK does not get properly disposed, and many errors surface when entering the new scene. This further means that all the memory allocated by the mrtk is not cleaned up when destroyed, which means that it leaks.

If the new scene contains an MRTKInstance, then errors prevent the scene from running properly.
If it does not and I instantiate an instance, the same errors appear.

If I set the MRTK instance to DontDestroyOnLoad, then objects the MRTK Instance created will have been destroyed, and cause errors. Notably created pointers will have been destroyed, and code using them will throw missing references exceptions.

Without using DontDestroyOnLoad, which I understand to be unsupported, I get the following errors:

`ArgumentNullException: Value cannot be null.

Parameter name: _unity_self
Microsoft.MixedReality.Toolkit.Windows.Input.GestureRecognizerExtensions.UpdateAndResetGestures (UnityEngine.XR.WSA.Input.GestureRecognizer recognizer, UnityEngine.XR.WSA.Input.GestureSettings gestureSettings) `

in GestureRecognizerExtensions.cs on line 11
bool reset = recognizer.IsCapturingGestures();

This seems to be due to the fact that the gesture recognizers are static instances, and when they are disposed they are never set to null, in order to be cleaned up by the garbage collector.

Next we have
NullReferenceException: Object reference not set to an instance of an object Microsoft.MixedReality.Toolkit.Input.BaseGenericInputSource..ctor (System.String name, Microsoft.MixedReality.Toolkit.Input.IMixedRealityPointer[] pointers, Microsoft.MixedReality.Toolkit.Input.InputSourceType sourceType on line 25

It seems the GazeProvider's Gazepointer instance is null, and is never properly recreated.

This error appears twice consecutively. When I did some debugging, it seems this is related to Speech and Dictation providers not being properly cleaned up.

To reproduce

Steps to reproduce the behavior:

1. Open a scene with an MRTK instance
2. Load a new scene with a different MRTK instance (or even a copy of the same one)
   2 ALT. Load a new scene and then at some later time instantiate an MRTK instance
3. Watch the errors poor in, which will mean nothing works.

Expected behavior

It should be possible to destroy the MRTK instance, when the MRTK instance is destroyed by a scene change, it should be fully cleaned up and not leak anything into the new scene.

It should be possible to change scenes and make use of different MRTK instances in each scene without having to keep a base scene with a persistent MRTK instance, and then carefully manage loading/unloading of scenes.

Your Setup (please complete the following information)

• Unity Version 2018.3.8f1 && 2019.1.3f1
• MRTK Version 2 - latest dev branch

Target Platform (please complete the following information)

• HoloLens
• HoloLens 2
• WMR immersive
• OpenVR

[provencher]

Speaking to @Railboy, I was informed that the currently supported scene switching flow is the following:

• Have a base scene with the MRTK, and then additively load your target scene, and unload it before loading a new one

This flow is not a standard flow in Unity development, and shouldn't be a requirement to working with the MRTK, as many projects will use traditional sceneloading without worrying about manually unloading loaded scenes before loading new ones.

I understand the new scene management service will facilitate this process, but it shouldn't be a requirement to work this way.

[keveleigh]

Some additional discussion has been had on https://github.com/microsoft/MixedRealityToolkit-Unity/issues/3599 and https://github.com/microsoft/MixedRealityToolkit-Unity/issues/3261

[Railboy]

@provencher I see to two related but ultimately separate issues being discussed here.

First is that the MRTK instance teardown isn't happening correctly. Definitely agree that if you destroy an MRTK instance neither it nor its core services should cause errors.

Second is our choice to support the practice of using a manger scene + additive loading. This is the workflow recommended by Unity and it's the one we'll be guiding people towards using.

If we solve the first problem then single scene loading shouldn't be an issue even if we don't officially support that workflow. More discussion here: #4089, #3987

IMO we should tackle this by writing some playmode tests that destroy the MRTK instance in a variety of reasonable ways (scene loads included) and then pick off the errors one by one.

[provencher]

@Railboy

For issue 1: You're correct, the root of all my issues is that the MRTK instance doesn't properly tear down.

For issue 2: The workflow you pointed to is for multi-scene EDITING in Unity, notably with large projects that require many scenes open at once. That link does not state that this is the way Unity recommends users make use of scene management in builds, and it certainly doesn't call for having a persistent base scene.

While having a persistent base scene is a valid way of working, it is by no means a standard nor recommend workflow with Unity, as far as I know, and in fact, knowing what scene an object belongs to is non-trivial. For that reason alone, I think single scene workflows, with non-additive scene loading, should be an officially supported workflow for the MRTK.

Finally, I think playmode tests are a good way to test this. Happy to assist with the testing.

[Railboy]

That link does not state that this is the way Unity recommends users make use of scene management in builds, and it certainly doesn't call for having a persistent base scene.

@provencher They reference this scenario further down in the page:

It is recommended to avoid using DontDestroyOnLoad to persist manager GameObjects that you want to survive across scene loads. Instead, create a manager scene that has all your managers and use SceneManager.LoadScene(, LoadSceneMode.Additive) and SceneManager.UnloadScene to manage your game progress.

But again, this is a separate issue so I don't want to fall down a rabbit hole. Top priority is instance teardown. With that in hand you can safely ignore our workflow suggestions.

[provencher]

Ah I missed that line, thanks for specifying.

Certainly, teardown is the priority.

[ywang530]

Speaking to @Railboy, I was informed that the currently supported scene switching flow is the following:

• Have a base scene with the MRTK, and then additively load your target scene, and unload it before loading a new one

This flow is not a standard flow in Unity development, and shouldn't be a requirement to working with the MRTK, as many projects will use traditional sceneloading without worrying about manually unloading loaded scenes before loading new ones.

I understand the new scene management service will facilitate this process, but it shouldn't be a requirement to work this way.

Hi I am developing an educational App using MRTK and having issues with scene changes. Can you be a bit more specific about how this flow works? So right now I have my main start menu scene with MixedRealityToolkit and Camera, and if I want to go to next scene I need to unload this current start menu scene and load the next scene additive? Is this what you are suggesting? For the "next scene" I want to load, should I include MixedRealityToolkit and Camera in there or not ?
Thanks in advance !

[Railboy]

So right now I have my main start menu scene with MixedRealityToolkit and Camera, and if I want to go to next scene I need to unload this current start menu scene and load the next scene additive?

@ywang530 You would start by loading a scene with a MixedRealityToolkit, Camera and anything else you want to exist for the lifetime of the app. And then you would never un-load it. Anything that you want created or destroyed over time would be kept in separate scenes and loaded / unloaded additively while the manager scene remains in place.

For the "next scene" I want to load, should I include MixedRealityToolkit and Camera in there or not ?

It doesn't matter - when you additively load a scene with a new MixedRealityToolkit, it will be disabled and ignored. Same with a new main camera.

The goal of the additive workflow is to ensure that your MRTK services aren't interrupted and that the objects they create (pointers, etc) aren't destroyed during scene loads. So the workflow isn't necessary if you don't care about services starting from scratch each time you load a scene. If you just want to load a bunch of standalone scenes one-by-one, you can put a MixedRealityToolkit and Camera in each scene and load them singly. Once the teardown issue is resolved, this will work fine.

We're working on docs / demos for both of these approaches.

[ywang530]

So right now I have my main start menu scene with MixedRealityToolkit and Camera, and if I want to go to next scene I need to unload this current start menu scene and load the next scene additive?

@ywang530 You would start by loading a scene with a MixedRealityToolkit, Camera and anything else you want to exist for the lifetime of the app. And then you would never un-load it. Anything that you want created or destroyed over time would be kept in separate scenes and loaded / unloaded additively while the manager scene remains in place.

For the "next scene" I want to load, should I include MixedRealityToolkit and Camera in there or not ?

It doesn't matter - when you additively load a scene with a new MixedRealityToolkit, it will be disabled and ignored. Same with a new main camera.

The goal of the additive workflow is to ensure that your MRTK services aren't interrupted and that the objects they create (pointers, etc) aren't destroyed during scene loads. So the workflow isn't necessary if you don't care about services starting from scratch each time you load a scene. If you just want to load a bunch of standalone scenes one-by-one, you can put a MixedRealityToolkit and Camera in each scene and load them singly. Once the teardown issue is resolved, this will work fine.

We're working on docs / demos for both of these approaches.

Thanks ! I tried to load the scene singly but somehow if I changed to another scene, I lost my controllers. So I have two scenes A and B and both contains a Camera and a MixedRealityToolkit. If I switch from scene A to scene B using SceneManager.LoadScene(Scene B, LoadSceneMode.Single); After I switch to scene B, Camera works fine but my both touch controllers are gone.

So I am using the Additive mode as you suggested. Everything seems working now.
Thank you so much for the explanation! Cant'wait to see the docs / demos.

[daveffs]

Further to this, I would like support for non-additive scene loads.

Our current flow is

• load bootstrap scene which contains MRTK and our manager objects
• load main scene - we currently rely on DontDestroyOnLoad on our managers, although we could change this to loading additively
• on a restart, we just load back to the main scene

For this to work with MRTK, we need to DontDestroyOnLoad the MixedRealityPlayspace and the UIRaycastCamera, otherwise they will be destroyed.

Discussed with @TobiahZ at Hololens2 hack week.

[provencher]

@daveffs

Don't use don't destroy on load.
Everything loaded in the bootstrap scene will exist in that scene.

The bootstrap scene should auto load your menu scene, and when you move passed that, you simply have to unload the menu scene before loading the new scenes.

Everything in the bootstrap scene will persist.

[Yoyozilla]

looks like this issue is fixed. Closing older issues. please reopen if repros in the latest release.

[julenka]

Reopening as this still does not work, I believe @davidkline-ms is working on a fix.