Filament: glTF support is missing features

[romainguy]

Our glTF viewer doesn't work entirely just yet. Here's what I found so far:

• Tangents/bitangents are not computed correctly if not present in the model
• We always use the default sampler instead of using the sampler described in the glTF file
• Vertex colors are ignored
• Transforms (rotation in particular) are not always handled properly

I found these issues while trying to load various Sketchfab assets.

[romainguy]

Rotation issue: https://sketchfab.com/models/a4ca0e3864e143af9bf1dffc7fc8029b (you have to zoom into the black sphere to seen the scene). This model also seems to use vertex colors? Our render is definitely off.

Non-default samplers: https://sketchfab.com/models/94b24a60dc1b48248de50bf087c0f042 The tramway and some of the walls are not textured properly, most likely because we don't use the right sampler

None of the textures are loaded for this model: https://sketchfab.com/models/282f497334f64a589edee4e63ad7e428

[romainguy]

Another bug: the water in https://sketchfab.com/models/e051a32d21d6453c8fad408082492267 is not transparent. The alpha mask is also not working on the bushes in that scene.

[prideout]

I'm working on the first bullet in the original problem description (Tangents/bitangents are not computed correctly), which is exemplified by glTF-Sample-Models/2.0/NormalTangentTest.

[magicwhale]

Rotation issue: https://sketchfab.com/models/a4ca0e3864e143af9bf1dffc7fc8029b (you have to zoom into the black sphere to seen the scene). This model also seems to use vertex colors? Our render is definitely off.

Non-default samplers: https://sketchfab.com/models/94b24a60dc1b48248de50bf087c0f042 The tramway and some of the walls are not textured properly, most likely because we don't use the right sampler

None of the textures are loaded for this model: https://sketchfab.com/models/282f497334f64a589edee4e63ad7e428

The first model seems to just have some rotation issues with the asset itself, and some of the colors aren't showing up because vertex colors aren't implemented yet.
For the second and third models, the issue seems to just be with the use of multiple UV coordinates.

Another bug: the water in https://sketchfab.com/models/e051a32d21d6453c8fad408082492267 is not transparent. The alpha mask is also not working on the bushes in that scene.

I realized that I didn't implement alpha masking and blending for unlit materials, which this scene uses.

I'm working on fixing the issues now.

[romainguy]

Crash opening this file: https://sketchfab.com/models/f8d3c7506f3545108dcf7f6c170e19e1

[magicwhale]

It seems like that file and the files mentioned in #503 have some kind of error within the gltf file, and this is causing the viewer to crash within Assimp when it is loading the gltf.

I saw that Don McCurdy's gltf viewer is still able to show the model even though it says there are errors within the file, so we could find a way to load the files by ignoring the malformed lines.

[donmccurdy]

It seems like that file and the files mentioned in #503 have some kind of error within the gltf file, and this is causing the viewer to crash within Assimp when it is loading the gltf.

If the errors seem unreasonable, I'm glad to send feedback to Sketchfab. Some viewers (like Facebook 3D Posts) enforce a "no validation errors" policy, and if these are issues you'd rather not work around, requiring stricter conformance from exporters is acceptable. The tolerance in my viewer is more accidental than planned. :)

[romainguy]

Well crashing is never a good option :) But if the crash is inside Assimp our best bet would be to fix it there and upstream the change. We could also investigate TinyGLTF

[romainguy]

Multi UV sets support is now in. @magicwhale were you able to figure out where the other issues come from?

[cdata]

@romainguy we recently proposed testing rendering parity between <model-viewer> (based on Three.js / GLTF Loader) and Filament for a number of the Khronos sample glTFs (relevant PR). We found samples that reproduce a number of the issues you referenced here. I'm leaving some of our test results below in case they might help inform your work to fix the related issues:

Tangents/bitangents are not computed correctly if not present in the model

Related sample model: Normal Tangent Mirror Test

Filament result|Expected result
---------------|---------------
|

Vertex colors are ignored

Related sample model: Vertex Color Test

Filament result|Expected result
---------------|---------------
|

Transforms (rotation in particular) are not always handled properly

Related sample model: Texture Transform Test

When I saw the results of this test it rang a bell. I'm not sure if you meant texture rotations, but perhaps this is relevant.

Filament result|Expected result
---------------|---------------
|

[romainguy]

Thank you. Texture transforms are just not supported at all at the moment. We're moving to a different gltf loader to fix all these issues.

[romainguy]

I'm surprised by the tangents though, @prideout wasn't this fixed?

[prideout]

Just now reproduced the issue with Normal Tangent Mirror, I'm surprised too. Will investigate.

[prideout]

tl;dr:

For now I think I should simply modify our copy of MeshAssimp to negate the bitangent, thus moving our current workaround downward so that it doesn't affect the tangents-are-available case. This will be fixed in a more proper way in the new glTF loader that I'm working on.

details:

We render NormalTangentTest correctly (where tangents are computed via MeshAssimp) but we do not render NormalTangentMirrorTest correctly (where tangents are supplied in the model).

When MeshAssimp generates tangents and bitangents, it basically makes tangents follow texcoord U, and bitangents follow texcoord V. The resulting tangent vectors are fairly consistent with a couple popular viewers that I tried, but the bitangent vectors are not. Other viewers use different approaches:

• ThreeJS (and therefore Don McCurdy's viewer) does this
• The Khronos ref viewer does this
• The spec recommends mikktspace (which Blender uses I think) which does this

[donmccurdy]

ThreeJS (and therefore Don McCurdy's viewer) does this

Note that this change is very recent, and won't be visible in my viewer until threejs r102 is released. But I think BabylonJS (and this viewer) do the same.

[romainguy]

Closing this for now. There are still a couple of unsupported features like blend shapes.