Three.js: GLTFLoader: metal-rough demo has sparkles ✨

[donmccurdy]

Carrying over conversation from #11088.

The THREE.GLTF2Loader BoomBox demo shows unexpected sparkles, possibly related to the textures. Also note that I've run Imagemagick on those textures (swapping r and b channels), so it's possibly an artifact of that.

[donmccurdy]

But I'm also able to reproduce similar issues importing several of BabylonJS's sample models, which have also been updated with the r→b change. The effect is especially noticeable on the lantern, if I remember right. There are a couple variations (png, jpg, jpg-with-quantized-png) we could test.

/cc @takahirox

[takahirox]

If I'm right, it'd be roughness related?

I removed maps and set roughness=0 in the example, it still has sparkles.
I'll take a look closer...

var mesh = scene.children[ 4 ].children[ 0 ].children[ 0 ];
var material = mesh.material;

material.map = null;
material.aoMap = null;
material.bumpMap = null;
material.emissiveMap = null;
material.envMap = null;
material.metalnessMap = null;
material.normalMap = null;
material.roughnessMap = null;

material.color.setRGB( 0, 0, 0 );
material.emissive.setRGB( 0, 0, 0 );

material.metalness = 1;
material.roughness = 0;

material.needsUpdate = true;

[WestLangley]

It appears to be an artifact of MeshStandard/Physical material.

It appears on other models, too -- especially where the surface has significant curvature. The artifacts can be seen even if there are no textures.

What is being seen are specular highlights, which for the GGX model appear to be very intense -- even for low-intensity lights.

[mrdoob]

/cc @bhouston

[bhouston]

I am not sure this is incorrect.

[bhouston]

Making metallic 1 and roughness 0 you have made a perfect mirror. It will reflect perfectly. The reflections will be intense from point light sources. Maybe do not use point light sources?

[bhouston]

Please post an interactive example?

[bhouston]

There are very serious issue with our or shader on mobile. I believe it is precision related but I haven't tracked it down fully get. The issue is related to the non metallic path - thus the diffuse brdf calculation. I think these sparkles are unrelated to those issues I am tracking.

[takahirox]

I've experimented.
It seems like D_GGX() in bsdfs.glsl causes these sparkles.

float D_GGX( const in float alpha, const in float dotNH ) {

    // alpha = pow2( clamp( roughnessFactor, 0.04, 1.0 ) );
    float a2 = pow2( alpha );

    float denom = pow2( dotNH ) * ( a2 - 1.0 ) + 1.0; // avoid alpha = 0 with dotNH = 1

    return RECIPROCAL_PI * a2 / pow2( denom );

}

https://github.com/mrdoob/three.js/blob/346f38c566a7291c4b2ef9f8e74198666770e2bb/src/renderers/shaders/ShaderChunk/bsdfs.glsl#L80-L88

Maybe the sparkles are the points where denom is close to zero?

I'll take a look closer later...

@bhouston

About example, you can see webgl_loader_gltf2.html example in dev branch.
Or you need online example?

[bhouston]

Maybe the sparkles are the points where denom is close to zero?

We clamped that before to get rid of divide by zeros because of rounding issues. Maybe it wasn't enough. Maybe we can just clamp were we do the divides?

Something like:

return RECIPROCAL_PI * a2 / max( pow2( denom ), 0.001 );

If we can understand the range for each variable in these mathematical equations, maybe we can reformulate the order of the math operations (at the cost of clarify of course) to avoid getting any intermediate result that is very small or very big. This would maintain the precision better.

For example, I notice that a2 is squared and so is the demoninator. Maybe we could do the divide before squaring them, and only square after the result? This would drastically reduce the required precision for the intermediate values.

[takahirox]

I've looked into more.

Maybe this issue(?) happens only when roughnessFactor is close to zero.

The sparkles may be the points where dotNH is close to one.

D_GGX is very sensitive to dotNH if roughnessFactor is close to zero and dotNH is close to one.
The following data set and chart is the result where roughnessFactor = 0.04 (minimum value).
(If roughnessFactor is large enough, D_GGX isn't so sensitive to dotNH)

dotNH     D_GGX
0.9980    0.050966531679918915
0.9981    0.05646315732146142
0.9982    0.06290010861631623
0.9983    0.07050481684920884
0.9984    0.07957787865714051
0.9985    0.09052323281265603
0.9986    0.10389392295773262
0.9987    0.12046337414117088
0.9988    0.1413398361248595
0.9989    0.16815667743808868
0.9990    0.2034019380878367
0.9991    0.25101706746464575
0.9992    0.31754893860041383
0.9993    0.41452709078025685
0.9994    0.5638179058765522
0.9995    0.8111257409349361
0.9996    1.2656408072680168
0.9997    2.245026411238407
0.9998    5.029413856561034
0.9999    19.862220229221098
1.0000    124339.79715663796

So precision can affect D_GGX a lot.

I tried some clamps and reformulates but they didn't work well on my environment.

Replacing a2 / pow2( denom ) with pow2( alpha / denom ) changes nothing.

Clamping return RECIPROCAL_PI * a2 / max( pow2( denom ), 0.001 ); removes the sparkles but
it seems it's too strong. I also tried some other clamps but they didn't work well neither.

Without the clamp.

With the clamp.

I'll try some more...

[bhouston]

Could you check the source paper for GGX and see if they discuss precision issues? Just a though. Also check the implementation of GGX in something like PBRT v3 and compare it with our implementation. While there is an integration in PBRT, the base equations are likely similar.

[WestLangley]

The sparkles may be the points where dotNH is close to one.

I do not think that is it.

I think it is just the specular reflection into the camera off of tiny facets. There is a lot of illumination in the scene -- probably because the model is so dark.

In the console type

scene.children[ 4 ].children[ 0 ].children[ 0 ].material.color.setRGB( 5, 5, 5 );

and the sparkles are less noticeable, even though the scene illumination is the same.

[bhouston]

@WestLangley wrote:

I think it is just the specular reflection into the camera off of tiny facets. There is a lot of illumination in the scene -- probably because the model is so dark.

I sort of agree with @WestLangley as I said before:
https://github.com/mrdoob/three.js/issues/11106#issuecomment-290985204

[takahirox]

This might not be from precision because nothing changes with presicion: 'lowp' for WebGLRenderer.

Ever I checked, I couldn't find any document about D_GGX which describe precision.

[takahirox]

I made an example with a simple black cylinder which reproduces a sparkle.
Slowly drag the mouse left and right to rotate the camera.

https://rawgit.com/takahirox/three.js/StandardMaterialSparkles/examples/standard_material_sparkles.html

Cylinder and material are

var geometry = new THREE.CylinderBufferGeometry( 0.1, 0.1, 0.5, 128 );
var material = new THREE.MeshStandardMaterial();
mesh = new THREE.Mesh( geometry, material );
material.color.setRGB( 0, 0, 0 );
material.metalness = 1;
material.roughness = 0;

[bhouston]

I'm not convinced this is a bug still. The reason is the sparkle looks like an aliasing issue. The highlight is very bright and very thin -- on my 4K monitor it is either one or two pixels wide -- and as I rotate it it switches between one and two pixels in width - thus shimmering. This is standard aliasing issues.

Usually you see pictures like this:
http://mielliott.github.io/Presentation/antialiasing-methods.png

The above picture is just a lot of sub-pixel features that if you only do a single sample per pixel to the underlying features you will get shimmers. I think this is similar to what you are seeing here.

To test if this is the issue, use the SSAARenderPass I contributed and set it to 32 samples. It will be brutally slow on anything but very high end video cards, but if the shimmer goes away, then it was likely an aliasing issue.

Besides full-screen SSAA, you can ensure you are using hardware-based MSAA and use MSAA WebGL2 render buffers or ensure you only render to the front buffer in WebGL 1. One could modify the BRDF to do some importance sampling techniques -- I'd recommend some type of Hammersley approach. Or you could limit roughness....

[bhouston]

It would be great to get optional (because it will be slower) importance sampling into our GGX and BlinnPhong BRDF implementations. One could configure the number of samples one wants to use -- and ensure the default of 1 sample is as fast as it is now. Here is a paper that describes importance sampling with, I believe, all of the math details required for implementation:

https://agraphicsguy.wordpress.com/2015/11/01/sampling-microfacet-brdf/

Importance sampling will produce the highest quality result for the required computational investment from the GPU. Better than ManualSSAA by far, and still better than hardware-based MSAA by at least twice if not more.

[bhouston]

More notes on importance sampling: http://blog.tobias-franke.eu/2014/03/30/notes_on_importance_sampling.html

Importance sampling is basically a way of anti-aliasing a BRDF when you have sharp highlights.

I suspect we could easily do 4 samples per pixel without any perceivable slowdown on low-end devices. I am not sure if 16 or 32 samples would be possible without a slowdown.

I wonder if we can meaningfully improve our results with just 4 importance-based samples per pixel? If we a random distribution I suspect 4 samples may lead to noise on sharp lightlights? Which may just be replacing one type of shimmer with another. Could one use structured sampling to avoid the random noise at low sample numbers? I do not know this area that well. I know that importance sampling narrows the sampling distribution as you decrease roughness, thus I guess you get the best use of those 4 samples -- so it may work okay?

Maybe just implement it and see if it works????

[bhouston]

@takahirox do you think you can implement importance sampling for GGX or BlinnPhong BRDFs?

[bhouston]

We should implement importance sampling for sampling from the IBL envMAp as well at low (oops I earlyer wrote high) roughness as I've seen sparkles there as well.

[takahirox]

@bhouston I wanna try!

[WestLangley]

@takahirox Can you please make the following changes to your demo and identify where you see sparkles.

camera = new THREE.PerspectiveCamera( 45, window.innerWidth / window.innerHeight, 0.001, 10 );

geometry1 = new THREE.CylinderBufferGeometry( 0.1, 0.1, 0.5, 256 );
geometry2 = new THREE.CylinderBufferGeometry( 0.1, 0.1, 0.5, 8 );

[bhouston]

BTW @spidersharma03 mentioned there is actually a field of study regarding "specular aliasing". Some publicatios that turned up when I searched this:

http://marcel-schindler.weebly.com/blog/specular-anti-aliasing
http://blog.selfshadow.com/2011/07/22/specular-showdown/

[Mugen87]

Relevant issue: #8183

[takahirox]

@WestLangley

geometry1
http://takahirox.github.io/sparkles/examples/standard_material_sparkles.html

geometry2
http://takahirox.github.io/sparkles/examples/standard_material_sparkles2.html

It looks same or similar sparkles on the same place.

@bhouston

Black cylinder with SSAARenderPass(it seems 16 samples is good enough).
No blinks.
http://takahirox.github.io/sparkles/examples/standard_material_sparkles_ssaa.html

glTF BoomBox with SSAARenderPass.
Less sparkles with 32 samples, but a little bit of them are left?
http://takahirox.github.io/sparkles/examples/webgl_loader_gltf2_ssaa.html

[WestLangley]

@takahirox Hmm..., I am not seeing any sparkles on the cylinders -- just a single specular highlight, which becomes circular( as is should be ) when zooming in. Perhaps you can provide a snapshot of what you are seeing?

( I should point out, you have an edge of the cylinder facing the camera, but rotating the cylinder geometry a bit has no effect as far as I can see.)

[takahirox]

Yup, it's circle if you zoom in but it blinks and is tiny on the default camera distance.
I think that's very similar to the BoomBox example so
the cylinder example reproduces the issue, am I wrong by any chance?

(GLTF2Loader example with 0.01 near camera)
http://takahirox.github.io/sparkles/examples/webgl_loader_gltf2.html

[mrdoob]

Related, I think...

http://jp.square-enix.com/tech/library/pdf/Error%20Reduction%20and%20Simplification%20for%20Shading%20Anti-Aliasing.pdf

[donmccurdy]

In the current example (https://threejs.org/examples/?q=gltf#webgl_loader_gltf_extensions), I don't see anything remotely like the issues of https://github.com/mrdoob/three.js/issues/11106#issuecomment-296409346, and we haven't had user reports. I think we could close this.