Three.js: Intent to implement/contribute Enterprise PBR improvements

[bhouston]

Description of the problem

To give context to a series of PRs are we are making to Three.JS I wanted to explain. We are adopting Enterprise PBR (PBR Next) as it is also the new material model that glTF is standardizing on. Enterprise PBR is a unification of advanced PBR parameters beyond just roughness and metalness.

It is specified here:
https://dassaultsystemes-technology.github.io/EnterprisePBRShadingModel/spec.md.html

The main additions to the Three.JS PBR model to adopt correctly Enterprise PBR are:

• [x] clear coat normal (see: https://github.com/mrdoob/three.js/issues/12867)
• [x] anisotropy that works under IBL (via bend normals) (see: https://github.com/mrdoob/three.js/issues/9250)
• [x] transparency (see: https://github.com/mrdoob/three.js/issues/15941)
• [x] sheen (see: https://github.com/mrdoob/three.js/pull/16971)
• [ ] reflectivityColor, which is the combined representation of IOR for F0, plus specular intensity and specular color (it means changing reflectivity from scalar to color: https://github.com/mrdoob/three.js/blob/dev/src/materials/MeshPhysicalMaterial.js#L21)
• [ ] subsurface model (attenuation color, attenuation distance, subsurface color - probably some type of improvement over: https://github.com/mrdoob/three.js/pull/13511 )

We are aiming to add these to the Node graph material system as I personally believe that is the future of material definitions (https://github.com/mrdoob/three.js/issues/16440). I think it is relatively easy to back-port them to PhysicalMaterial as well.

This will be useful for an advanced glTF loader in Three.JS.

I believe it is also the direction other projects are going, including Google's Filament. We are aiming to have our Three.JS contributes compatible with Filament directly when possible.

Three.js version

• [x] Dev

Browser

• [x] All of them

OS

• [x] All of them

Hardware Requirements (graphics card, VR Device, ...)

[bhouston]

/ping @DanielSturk - this gives context to your PRs so that people understand the overall motivation.

[donmccurdy]

Enthusiastic +1 from me! 🙂

This link is currently broken, but the <model-viewer/> project has some render comparison infrastructure (for three.js and Filament) that may be helpful along the way.

[elalish]

Enthusiastic +1 from me!

This link is currently broken, but the <model-viewer/> project has some render comparison infrastructure (for three.js and Filament) that may be helpful along the way.

No longer broken!

[donmccurdy]

The glTF sheen extension (KHR_materials_sheen) is nearly complete, although still welcoming feedback for a bit longer. To implement in MeshPhysicalMaterial, it would require:

• sheen
• sheenMap
• sheenRoughness
• sheenRoughnessMap

The two maps use .rgb and .a channels respectively; they can be combined.

[donmccurdy]

The glTF extensions KHR_materials_sheen and KHR_materials_transmission are now complete. Would be great if we could work toward supporting the remaining properties of those PBR features. See this article for a bit more info.

[rawwerks]

+1 for this, in particular a KHR_materials_transmission implementation would have a massive impact on rendering physical products.

i have no appreciation for how much work this is, but i would be curious @mrdoob if this is in line w/ your near-term roadmap for three.js?

re: implementation, the only support i can offer is that both https://github.com/BabylonJS and https://github.com/KhronosGroup/glTF-Sample-Viewer have successfully implemented all three of the new PBR extensions - so perhaps there are methods that can be borrowed from those projects.

also - the new Khronos "toy car" is an all-in-one test of a successful implementation: https://github.com/KhronosGroup/glTF-Sample-Models/tree/a35e94effc01db54f94bab34f793c960276a67fc/2.0/ToyCar

[mrdoob]

How should this be architected? Seems to me that KHR_materials_transmission would require this:

1 - Render opaque to a render target (front to back)
2 - Render transparent to a render target (back to front)
3 - Render refractive using the current render target (back to front)

Considering that WebGL1 doesn't support multisampled render targets we can't just use this architecture for everything. WebGL1 people will see either things aliased or things not refracting.

This could be the default architecture for WebGL2 though.

[bhouston]

The proposed suggestion only works for single layered transmission. I believe the more correct approach would be to implement an OIT method. OIT = order independent transparency, per https://github.com/mrdoob/three.js/issues/9977 It would then work on WebGL1 and 2.

Cesium project has OIT code: https://gitlab.sensoro.com/wushijing/cesium/blob/9fd4154a2eb3696f1c4c053ccf3a9b8354d683d4/Source/Scene/OIT.js. Clara.io has also implemented OIT.

https://cesium.com/blog/2014/03/14/weighted-blended-order-independent-transparency/

[mrdoob]

I'm not sure OIT solves refraction.

[rawwerks]

I might be mis-reading the documentation, but I believe that the KHR_materials_transmission extension solves the refraction part regardless of OIT: https://github.com/KhronosGroup/glTF/tree/master/extensions/2.0/Khronos/KHR_materials_transmission#refraction

_Therefore, correct ordering is not an absolute requirement when implementing this extension in realtime renderers, nor is rendering all potentially overlapping layers._

Is this example from the GLTF Sample Viewer (which also uses WebGL) relevant to the challenge here?

Transmission set from extension https://github.com/KhronosGroup/glTF-Sample-Viewer/blob/2e6f9f1cfef04239cc8c8c403a5c49a242b1dc3f/src/material.js

```
// KHR Extension: Transmission
if (this.extensions.KHR_materials_transmission !== undefined)
{
let transmission = this.extensions.KHR_materials_transmission.transmission;

            if (transmission === undefined)
            {
                transmission = 0.0;
            }

            this.defines.push("MATERIAL_TRANSMISSION 1");

            this.properties.set("u_Transmission", transmission);
        }

then loaded into PBR shader https://github.com/KhronosGroup/glTF-Sample-Viewer/blob/2e6f9f1cfef04239cc8c8c403a5c49a242b1dc3f/src/shaders/pbr.frag

#ifdef MATERIAL_TRANSMISSION
vec3 diffuse = mix(f_diffuse, f_transmission, materialInfo.transmission);
#else
vec3 diffuse = f_diffuse;
#endif

color = (f_emissive + diffuse + f_specular + f_subsurface + (1.0 - reflectance) * f_sheen) * (1.0 - clearcoatFactor * clearcoatFresnel) + f_clearcoat * clearcoatFactor;

```

index of refraction is set by extensions.KHR_materials_ior and used here: https://github.com/KhronosGroup/glTF-Sample-Viewer/blob/2e6f9f1cfef04239cc8c8c403a5c49a242b1dc3f/src/shaders/ibl.glsl , which contains getIBLRadianceTransmission that is then used in 'pbr.frag'

@MiiBond - any ideas?

[MiiBond]

Hello. Yeah, OIT is about rendering transparent polys in the correct order (or approximately correct order in some cases). The most important requirement for the transmission extension is rendering transparency with the correct blending and refraction. Transmissive surfaces can both absorb and reflect light so it's not really possible to use traditional OIT methods as two different blend modes are needed. Also, refraction requires sampling from an already-rendered target.
What we did for the Babylon loader was:

1. Set up a target where only opaque objects are rendered in a first pass (this can be slightly lower resolution than the canvas and power-of-two dimensions so that mips can be generated every frame).
2. The whole scene is then rendered normally and transmissive materials use the opaque render target to render when calculating refraction and absorption of the background scene. You can also use the mips to represent light transmitting through a rough surface.

This is very straightforward to do so it's what we suggested as the bare-bones approach to supporting the transmission extension. It's also, notably, what SketchFab seems to do for refraction. They also only support refraction of opaque objects.

If you want to get more complicated, you can combined some OIT techniques. I used depth-peeling, combined with MRT to render out a g-buffer (of sorts) and then composite them back together to render multiple layers of transparency with correct PBR blending. Here's an example scene:
https://adobe.ly/33DOPoA

[mrdoob]

@MiiBond Super helpful! Many thanks! 🙏