Mixedrealitytoolkit-unity: Gltf models exception on load

[edgarrodrigosantos]

Describe the bug

Got to a point where mrtk2 sample gltf model (avocado) is loaded correctly (had to use pre-release 2.3.0) on Hololens but our own gltf models don't.

To reproduce

Steps to reproduce the behavior:

1. Try to load the attached gltf
   assem1.zip

Expected behavior

The gltf model is loaded correctly.

Your setup

• Unity Version 2019.3.0f6
• MRTK Version v2.3.0

Target platform

• HoloLens (but can be reproduced in the editor)

Additional context

The exception is the following:

TestGltfLoading start failed - JSON parse error: The document root must not follow by other values.
at (wrapper managed-to-native) UnityEngine.JsonUtility.FromJsonInternal(string,object,System.Type)
at UnityEngine.JsonUtility.FromJson (System.String json, System.Type type) [0x0005c] in <090959ea8bda448d921c6cab7fb64dba>:0
at UnityEngine.JsonUtility.FromJson[T] (System.String json) [0x00001] in <090959ea8bda448d921c6cab7fb64dba>:0
at Microsoft.MixedReality.Toolkit.Utilities.Gltf.Serialization.GltfUtility.GetGltfObjectFromJson (System.String jsonString) [0x00170] in E:\KIT-AR\projects\19109-gltfloader\gltfExplorationB\GLTFExploration_B_2018\Assets\MixedRealityToolkit\Utilities\Gltf\Serialization\GltfUtility.cs:202
at Microsoft.MixedReality.Toolkit.Utilities.Gltf.Serialization.GltfUtility+d__4.MoveNext () [0x0011a] in E:\KIT-AR\projects\19109-gltfloader\gltfExplorationB\GLTFExploration_B_2018\Assets\MixedRealityToolkit\Utilities\Gltf\Serialization\GltfUtility.cs:62

[wiwei]

So it looks like what's going on here is a limitation of Unity's JsonUtility in parsing out the .gltf that's included there (i.e. as it turns out Unity's JSON parsing isn't really a general purpose thing, it's really specifically for handling Unity's serialization requirements)

This is tricky because AFAICT the other option here is really to pull in a real JSON loader (i.e. Newtonsoft) but doing so also requires some trickiness around how we pull in NuGet packages.

Internally within your own stuff, you could modify that code to use a different Json parsing library (i.e. newtonsoft), while we figure out how best to disentangle some of this.

[keveleigh]

It also might be worth running your model through http://github.khronos.org/glTF-Validator/, to double check that the format is correct.

[edgarrodrigosantos]

I tried changing to Newtonsoft parsing but still get an error related to the expected schema (I assume):
TestGltfLoading start failed - Error converting value "attributes" to type 'Microsoft.MixedReality.Toolkit.Utilities.Gltf.Schema.GltfMeshPrimitiveAttributes'. Path '', line 1, position 12.
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType (Newtonsoft.Json.JsonReader reader, System.Object value, System.Globalization.CultureInfo culture, Newtonsoft.Json.Serialization.JsonContract contract, System.Type targetType) [0x000bd] in <1fa899c21f554672afbc630cc93b185e>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x000d7] in <1fa899c21f554672afbc630cc93b185e>:0

I run the gltf through the validator and have minor issues only (related to export data from SolidWorks). Can't seem to be able to use this to load our gltfs even though we can load them with other libraries...

[wiwei]

Ah... in that case, there could be an issue where where the GLTF schema here might have been updated since our initial drop of UnityGLTF. There might actually be some mismatch between the new UnityGLTF code and when we took the drop a while back.

In that case there are gonna more hoops for you to jump through until we get this GLTF sorted out (or you could try to grab the latest from the KhronosGroup's UnityGLTF)

[joaocc]

@wiwei Do you have any plans you can share on getting glTF support in MRTK2 more robust? Thx

[wiwei]

@joaocc this is something that we've debated internally a bunch, and I think we're mostly looking for signal here for how many folks are using this and if it's really valuable.

I think that we pulled this in for some of our own GLTF model loading but it causes issues when folks are pulling in MRTK where they already have their own GLTF loading code.

I think the most helpful part on our side is knowing how popular this is, or if we should actually try to detach the forked version and let folks bring their own (i.e. just use UnityGLTF directly)

[wiwei]

@edgarrodrigosantos also FYI in case you know of others who have opinions on how you want to consume GLTF stuff (through MRTK, or through UnityGLTF plugin itself)

[edgarrodrigosantos]

@wiwei The problem with UnityGLTF is that it doesn't work properly with Hololens builds. We had issues related to non-deterministic behavior loading materials/textures, problems loading some of our models, etc.... for comparison we're using a plugin (trilib) that loads all model ok but that is one more dependency that we would like to avoid and that is also not tailored to Hololens/UWP.

[wiwei]

@edgarrodrigosantos gotcha - do you have a pointer to issues on UnityGLTF (i.e. specific issues on their project) tracking the issues you've hit? I'm just curious on my end to know more about them.

[edgarrodrigosantos]

Unfortunately I still didn't have the time to open them (with proper examples demonstrating all the issues, some of the gltfs I cannot even share so it requires some time on my end).

But I will and I'll link them here.

EDIT: I quickly created a ticket with one of the issues, you can find it here:
https://github.com/KhronosGroup/UnityGLTF/issues/559

[Rosthouse]

I ran into a similar problem today and I think I've figured out what's going on. The problem is with parsing the "attributes" tag inside of a mesh primitive. As an example, take the following mesh primitive:

{
    "attributes" : {
         "POSITION" : 0,
         "NORMAL" : 1,
         "TEXCOORD_0" : 2
    },
    "indices" : 3
}

(the other values inside the GLTF are not relevant, so I ommited them)

Now in GltfUtility.GetGltfMeshPrimitiveAttributes, this attribute is parsed with a regex. If you check the json object closely, you notice that there is a space between "attributes" and the following colon. I believe that is also the issue with the GLTF provided by @edgarrodrigosantos, since I can also see additional spaces inside that file.

The problem is that the method executing this regex will try to do a string replacement for "attributes": (replacing it with an empty string). This of course can't happen, due to an additional space in the GLTF file.

A proposed fix would be to execute a regex not on "attributes"

"(?<Attributes>\"attributes\"[^}]+})"

but rather on the content of "attributes", like this:

"\"attributes\" ?: ?(?<Data>{[^}]+})"

(which would of course also require a change in the method GltfUtility.GetGltfMeshPrimitiveAttributes, adding the content of the matched string to the output.

[Rosthouse]

If need be, I would do a pull request for this issue :)

[keveleigh]

If need be, I would do a pull request for this issue :)

@Rosthouse Please feel free! It sounds like you have a good handle on the issue and a fix