Three.js: Adding support for the glTF 2.0 KHR_materials_variants extension

[sleroux]

Description of the problem

Facebook has been putting together a proposal for a new glTF 2.0 extension that will allow melding multiple material variants for a model into a single glTF/GLB: https://github.com/KhronosGroup/glTF/pull/1681

It's being proposed as a Facebook vendor extension at the moment with the eventual goal to promote it to an EXT extension. There hoping to start adding support for the extension across various 3D viewers and asset content creation tools. Having it be part of three.js would be a great way to reduce model file size - especially for commerce experiences where it can be common for models to share geometry but have different materials (ie. Selecting different colors of shoes).

@zellski @mikkoh @pushmatrix @richardmonette

Three.js version

• [ ] Dev
• [ ] r109
• [ ] ...

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

[donmccurdy]

Hi @sleroux!

Both yes and no. 😇

I believe FB_material_variants requires a level of abstraction that typically exists in applications built on three.js, but not in three.js itself. When three.js loads a glTF asset, it returns one or more native three.js objects (THREE.Scene, THREE.Mesh, ...). There is no three.js abstraction for variants, and it probably doesn't make sense for there to be one.

However, allowing applications and tools built on three.js to support FB_material_variants sounds very reasonable, and in fact we might already be there. Three.js supports lazy-loading materials from a glTF asset, and preserves the JSON data of unknown extensions so that applications can access it. So for a quick, untested suggestion, I believe something like this would work today:

loader.load('shoe.glb', ({scene, parser}) => {

  // Add asset to the scene.
  globalScene.add(scene);

  // Switch to 'gold_trim' variant.
  scene.traverse(async (object) => {
    if (!object.isMesh) return;
    const {mapping} = object.userData.gltfExtensions['FB_material_variants'];
    const variant = mapping.find((v) => v.tags.includes('gold_trim'));
    object.material = await parser.getDependency('material', v.material);
  });

});

Caching and reuse of pre-loaded materials is handled automatically in the parser API.

[sleroux]

Thanks for the quick reply! I wasn’t sure exactly where this type of extension would live with regards to the three.js stack so this was super helpful.

Three.js supports lazy-loading materials from a glTF asset, and preserves the JSON data of unknown extensions so that applications can access it.

I really like this approach for seeing how we can start combining the functionality of the extension with three.js. Would putting something together for this as a three.js example be the right level abstraction for this?

[donmccurdy]

That sounds about right — maybe something similar to https://threejs.org/examples/?q=car#webgl_materials_cars? I'm not sure whether we'd be ready to merge such an example right now, or would want to wait for a later EXT_ or KHR_ version, but in any case a proof of concept would be great. 🙂

[zellski]

Yeah, this is an interesting aspect of this extension – indeed, the markup by itself has no immediate bearing on how a model would be interpreted, and it's unlikely that a general purpose loader would want to be loaded down with these rather specialised concerns. A utility class or method or wrapper seems like a wiser idea.

That's not to say there aren't constraints on the loader (and the engine in which it lives). For example, materials that are only referenced in variant extension information, but not in glTF vanilla – will they have been preloaded? Likely not. We have the lovely API call parser.getDependency() above, and a utility class could presumably execute those preemptively to avoid latency hiccups later. Related, variant-extended assets will likely make more use of shared resources than many other types of model, so it matters that an engine doesn't create copies of data that only exists as singletons in the data. A mature engine like three.js is quite helpful here (though I will admit I don't know the details of how well 1:N references are handled across the board.)

[arodic]

This issue should be renamed to KHR_materials_variants

[cx20]

I took @donmccurdy's advice and incorporated the display features of this extension into the gltf-test three.js example.
To get the userData property from mesh, need r123 or higher. See this PR https://github.com/mrdoob/three.js/pull/20679

See below for the results.
https://github.com/KhronosGroup/glTF-Sample-Models/pull/271#issuecomment-727074863

[donmccurdy]

Thanks @cx20! I think we should add an example (see https://github.com/mrdoob/three.js/issues/17808#issuecomment-546139113) but after that this issue can probably be closed.

[cx20]

I would like to add the KHR_materials_variants extension to the webgl_loader_gltf_extensions example.
I am concerned about the bloat of the repository by adding sample models.
The size of the model is a bit large, but I'd like to use the famous shoes as a sample for this glTF extension, is that OK?
As for the other samples, the chair samples are a little smaller in size.

Shoes : 7.47 MB (glTF-Binary)
https://github.com/pushmatrix/glTF-Sample-Models/tree/master/2.0/MaterialsVariantsShoe

Chair : 1.75 MB (glTF-Binary)
https://github.com/pushmatrix/glTF-Sample-Models/blob/master/2.0/MaterialsVariantsChair

[donmccurdy]

I'd suggest starting a new example — the source code for webgl_loader_gltf_extensions is already too long to be instructive for new users. Perhaps webgl_loader_gltf_variants? The shoe example could probably be optimized down to <2 MB. If that plan is OK with everyone, I can make a smaller version of the model.

[cx20]

@donmccurdy I'm going to make a separate example webgl_loader_gltf_variants.

I can make a smaller version of the model.

Can I ask you to minimize the glTF file?
I tried to resize it, but I couldn't find an exporter (or converter) that would support this extension.
After resizing the textures, I tried to save the .gltf as .glb in Babylon.js Sandbox, but the information in the KHR_materials_variants extension seems to be lost.

[donmccurdy]

I was a bit too optimistic — could only get it down to 3 MB, mostly because the normal map is very sensitive to compression and resizing. Here's the result:

MaterialsVariantsShoeCompressed.glb.zip

| before (7.5 MB) | after (2.9 MB) |
|---|---|
| | |

This does depend on KHR_texture_basisu and KHR_draco_mesh_compression, so you'll need to call setDRACOLoader and setKTX2Loader in the example. The KHR_materials_variants data should be intact, but let me know if anything looks incorrect.

[cx20]

@donmccurdy Thanks for the support for resizing.
I find it interesting to use the KTX2 and Draco extensions, but I fear it will be a bit complicated for a sample.

I tried resizing the texture manually by going from 2048x2048 to 1024x1024.
And I was able to shrink the base64 encoded and embedded mesh data from any amount by making it a .bin file.
The size of the glTF file is currently about 2.5MB, but its appearance has not changed much.

[cx20]

Opps, I thought it was working locally, but it seems I was testing with the cached version. I noticed that the hand-modified model does not display well, so I will try again.

[cx20]

I think the example is working now.

GLTF Loader + KHR_materials_variants result:

[cx20]

I find it interesting to use the KTX2 and Draco extensions, but I fear it will be a bit complicated for a sample.

I'm wondering if I should switch to the KTX2 compression version.
The reason is that KTX2 compression reduces the file size while maintaining the quality of the texture.

The changes to the code are as follows It doesn't appear to be that complicated.

<script src="./js/libs/basis/msc_basis_transcoder.js"></script>

import { DRACOLoader } from './jsm/loaders/DRACOLoader.js';
import { KTX2Loader } from './jsm/loaders/KTX2Loader.js';

const loader = new GLTFLoader();

const dracoLoader = new DRACOLoader().setDecoderPath( 'js/libs/draco/gltf/' );
loader.setDRACOLoader( dracoLoader );

const ktx2Loader = new KTX2Loader().detectSupport( renderer );
loader.setKTX2Loader( ktx2Loader );

[donmccurdy]

The new demo looks good to me, we can always refine it more later — thank you @cx20!