Three.js: GLTFLoader: Add accessibility to original JSON index from THREE objects

[FMS-Cat]

Hi!

Today I needed to determine what material in original glTF structure the converted material is.
The reason why I have to do that is, to convert materials generated from GLTFLoader to another one based on external index-property table.
But, I couldn't get how I can access to original index of the material.

Then, I added a feature to my fork of GLTFLoader that enables me to access original index of GLTF structure.
Here's a quick example of what I did:

And here's a diff I did to GLTFLoader.js : https://github.com/FMS-Cat/three.js/commit/acec47fde5bb2dfdece431ed50aa299102a6b7a8

I have 100% confidence there are so many better way to perform this but I'm rushing so here's an issue instead of PR:
I want to access to the index of each stuff (nodes, materials, meshes...) in original glTF structure from converted THREE entities.

[donmccurdy]

/cc https://discourse.threejs.org/t/how-to-use-a-gltf-material-library-concept/5323/2

Could you explain further what you're trying to do with the indices? There may be multiple materials in three.js generated from a single material in glTF (e.g. if a material is used on meshes with and without vertex colors), so indices may not correspond 1:1 here.

Another option would be to use the .name property in glTF and three.js. You can access the original glTF data from the GLTFLoader response:

loader.load('foo.glb', (gltf) => {
  console.log( gltf.parser.json );
});

[FMS-Cat]

Could you explain further what you're trying to do with the indices?

What I want to do here is to load additional material properties that is contained in extensions field of glTF.
However these extension properties are on root position of glTF, instead of materials[index].extensions e.g. json.extensions.VRM.materialProperties[materialIndex].
so I need to refer index numbers of converted (by GLTFLoader) materials in order to re-convert them.

Another option would be to use the .name property in glTF and three.js

True, and I used to do that previously, but it doesn't work since .name can be not unique (and it isn't invalid in glTF spec) 😢

Also I need to refer this issue: https://github.com/mrdoob/three.js/issues/11682
Make the GLTFLoader hookable may also solve the issue but yeah this plan looks like so exhaustive.

[donmccurdy]

Do you mean the extensions field of the material? Or something else? You should be able to see those additional properties in material.userData.*.

[donmccurdy]

Oops, no I was incorrect about that — three.js does not have .userData on materials, only Geometry, Mesh, and other Object3D-extended objects... if the contents of the glTF material's .extensions field were passed into the threejs material's .userData field, would that do what you need?

[FMS-Cat]

Three.js does not have .userData on materials

Nooooo I didn't know that!! I think it should though!

if the contents of the glTF material's .extensions field were passed into the threejs material's .userData field

Sadly no 😢 , since what I want to see is not
json.materials[materialIndex].extensions...
but
json.extensions.VRM.materialProperties[materialIndex]...
I'm so sorry about the spec is kinda uh but it's already well distributed format in there...
So I still need material indices to "hack" the glTF materials.

Anyway, containing extension field of glTF into THREE.Material.userData sounds really good!
Aside of my demand we should do this :joy:

[donmccurdy]

AH sorry again I misread and should have just checked the docs. material.userData is fine and supported now... https://github.com/mrdoob/three.js/blob/dev/examples/js/loaders/GLTFLoader.js#L2408-L2410

In the future I might suggest that VRM put its extensions directly onto the glTF material definitions, but I understand that's not a change that can be made right now.

For now, here are two ideas:

1. I'd like to have a 'lazy-loading' mode for GLTFLoader, in which you could do something like this:

var loader = new THREE.GLTFLoader();
loader.createParser( 'model.glb', ( parser ) => {
  parser.json.extensions.VRM.materialProperties.forEach( ( props, index ) => {
    parser.json.materials[ index ].extensions = { VRM: props };
  } );
  parser.getDependency( 'scene', 0 ).then( ( scene ) => {
    console.log( scene.children[0].material.userData.VRM ); // material has VRM data now
  } );
} );

1. Adding the material's index to its .userData, as you suggested.

/cc @robertlong this seems relevant to some of what you've been doing, any thoughts?

[FMS-Cat]

material.userData is fine and supported now...

🙌

In the future I might suggest that VRM put its extensions directly onto the glTF material definitions

Yes I should! I'll blame that later.

I'd like to have a 'lazy-loading' mode for GLTFLoader

The code looks cool! I felt it's far better than my idea.

[takahirox]

1. is what I plan to do in VRMLoader once we introduce .createParser() (or something) to GLTFLoader.

[takahirox]

FYI, I suggested putting the extensions under the leaf like material instead of under the root before but they don't they do (so far). https://twitter.com/MobileHackerz/status/998508907386421250 (tweet in Japanese).

[mrdoob]

@takahirox I think @FMS-Cat is writing a VRMLoader 😉

[takahirox]

Ah, he seems working for VRoidHub

[takahirox]

@FMS-Cat Do you plan to contribute to Three.js VRMLoader? Or is your loader VRoidHub specific?

[FMS-Cat]

revisited!

I'm thinking about https://github.com/mrdoob/three.js/pull/15508 , it can resolve my problem in real smart way

Do you plan to contribute to Three.js VRMLoader? Or is your loader VRoidHub specific?

We are still discussing about this, I definitely want to make it open-source though!

[takahirox]

We are still discussing about this

License issue? Or technical matter we can help?

[FMS-Cat]

I'm not gonna talk about our internal issues but talking of license we definitely should release that in MIT since both UniVRM and Three.js follows the way 🤗.

We are currently thinking about how we should let the entire workflow interface, since we might rather prepare a VRM class with a handful of methods than just an "interface".

[FMS-Cat]

digging up my old issue...

Now we have https://github.com/pixiv/three-vrm/ !
In this implementation, we addressed the challenge by using parser.getDependency in wiser implementation. It still is depending on the fact that the result of getDependencies('mesh') grabs the reference for the objects on my current scene though...

[donmccurdy]

In this implementation, we addressed the challenge by using parser.getDependency in wiser implementation.

Is this working OK for you? Or did you still need something like createParser()?

It still is depending on the fact that the result of getDependencies('mesh') grabs the reference for the objects on my current scene though....

That's a problem because if the mesh was used in multiple places, the meshes in your scene will have been cloned from that dependency? Or is there another problem?

[FMS-Cat]

did you still need something like createParser()?

It's working really fine right now, but yes, if we can have createParser, that still makes our implementation straightforward and easy to understand.

That's a problem because if the mesh was used in multiple places, the meshes in your scene will have been cloned from that dependency? Or is there another problem?

Since we don't have a detailed specification of GLTFLoader, we don't get the behavior of getDependency until we see the implementation, so utilizing the internal cache might be evil and confusing

[donmccurdy]

It's definitely intentional that the internal GLTFLoader cache will always return the same object when called with the same arguments, like "mesh", 0. That is what guarantees geometries and images are reused efficiently within a scene. You can rely on that, but I would not rely on those necessarily being the only meshes in the scene. For example, mesh "0" might have to be cloned (reusing the same geometry and material) if it's attached at multiple places in the scene.

[FMS-Cat]

Yes that is possible and if we get such an example on our library ours does not work properly...... :weary:

[donmccurdy]

@FMS-Cat this issue would be solved by https://github.com/mrdoob/three.js/pull/18421 I think, do you agree?

[FMS-Cat]

18421 sounds good! Yes this is a very valid solution for this

[donmccurdy]

Going to close this issue, since https://github.com/mrdoob/three.js/issues/11682 is really the same issue it seems like, and https://github.com/mrdoob/three.js/pull/18421 should fix both.