Three.js: Cannot see glass transparency on GLTF 3D model

[ranbuch]

Description of the problem

• [ ] Dev
• [x] r91
• [ ] ...

Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

OS

• [x] All of them
• [ ] Windows
• [ ] macOS
• [ ] Linux
• [ ] Android
• [ ] iOS

Here is a GLB file that contains a Glenfiddich bottle 3D model.
The bottle is transparent so you can see the liquid inside of it.

When viewing on babylonjs you can actually see the liquid inside but on threejs you cannot see the bottle transparency.

[looeee]

Are you sure that model is correct? I've tried loading it in a couple of different places, including the babylonjs playground, but they all give an error.

[ranbuch]

@looeee you are right, sorry about that.
here's a link to the correct one.

[takahirox]

I don't see the liquid inside even on Babylon.

[mrdoob]

When viewing on babylonjs you can actually see the liquid inside but on threejs you cannot see the bottle transparency.

Can you share screenshots?

[looeee]

If you zoom in quite a bit you can see it

And if you rotate up it suddenly jumps in

[takahirox]

OK, confirmed. This could be from depth test and render order...? I set 1 to .renderOrder of the bin and then I can see the liquid inside. If so, this isn't glTF specific issue. I'll take a look closer...

[takahirox]

This's my speculation.

With depth test and alpha blending, we need to render transparent object in order "back to front". But in that example, I guess the bin is rendered first and then the liquid is rendered because they're on the same position and the liquid mesh is generated later. Three.js calculates the distance from camera with object z position, not the surface.

I'll make an easy example later which can reproduce the error if my speculation is right.

[mrdoob]

I wonder what's babylon.js logic for sorting objects.

[takahirox]

I made an example

https://jsfiddle.net/ptgwhemb/186/

Two different size balls on the same position but the smaller one can't be seen. To see the smaller one, set biggerBall.renderOrder = 1; or generate the smaller ball first.

So as I mentioned, this issue isn't glTF specific but generic Three.js issue.

@mrdoob

Curious to know, too. I guess, calculating boundingBox/Sphere and rendering the smaller one first if two or more object on the same position?

[takahirox]

Let's change the subject of this issue. "Wrong rendering order of objects on the same position" or something.

[WestLangley]

Let's change the subject of this issue. "Wrong rendering order of objects on the same position" or something.

I would say that is a hypothesized cause, not the symptom. I think the title is fine. This symptom is just not glTF-specific.

Related: https://stackoverflow.com/a/13236863/1461008

If there is an engine that always renders this model correctly, then we should figure out how to, also. Otherwise, it is a modeling issue.

[donmccurdy]

I consider this a modeling issue, more discussion in https://github.com/KhronosGroup/glTF/issues/822. glTF represents only PBR and alpha coverage, not true transparency; things like glass or water require other material models. A content author will need to understand those issues and make tradeoffs to get the results they want.

We'll eventually have a glTF extension for blend modes in which case a multiplicative and additive pass can be used for order-independent glass rendering: https://poly.google.com/view/atB26Z6BPd0

[takahirox]

OK, I searched a bit and seems like sorting for transparent object which contains another transparent object, like the bin and the liquid in that model, is a known generic problem. So agreed with a modeling issue.

BTW, I'm not familiar with order independent transparency so curious. How can multiplicative and additive pass solve this issue? With turning depth write off? How does glTF specify depth write off material?

[donmccurdy]

glTF doesn't specify depth write settings, no, in any case this requires some work from the author.

[takahirox]

Why did you introduce glTF blend extension here? I don't think soly it can solve this issue. I'm a bit confused.

[donmccurdy]

I just mean to say that transparent glass surrounding a solid object is not as simple as setting alpha < 1.0. This is not something we can fix by changing GLTFLoader.

[WestLangley]

I think we agree this is a modeling issue, so closing.

[ranbuch]

@WestLangley from what I understood that's a wrong rendering order bug witch babylonjs has resolved.

[WestLangley]

As I said,

If there is an engine that always renders this model correctly, then we should figure out how to, also. Otherwise, it is a modeling issue.

As @takahirox said

I don't see the liquid inside even on Babylon.

[ranbuch]

As @takahirox said:

OK, confirmed.

As @mrdoob said:

I wonder what's babylon.js logic for sorting objects.

As @donmccurdy said:

in any case this requires some work from the author.

@WestLangley I am under the impression that in the beginning you guys thought there's no issue but changed your mind afterwards.

[donmccurdy]

In BabylonJS, the liquid seems to flicker in and out depending on the viewing angle; this is not an issue that the engine can fix automatically with our current method of handling transparency.

@ranbuch see this question about transparent objects in three.js on stack overflow, and in particular the suggestions about .sortObjects, .depthWrite, and .renderOrder.

[ranbuch]

That's kind of a longshot but I've modified my local three.js file (r92) and it seems to solve this issue.

On line 17,162 I've altered the function reversePainterSortStable in the next why:

function reversePainterSortStable( a, b ) {

        if ( a.renderOrder !== b.renderOrder ) {

            return a.renderOrder - b.renderOrder;

        } if ( a.z !== b.z ) {

            return b.z - a.z;

        } else {
            // In case the objects are of type Mesh we'll sort by mesh size
            // that why we might solve the transparency render order issue
            if (a.object && a.object.isMesh && b.object && b.object.isMesh )
                return getObjectSize(a) - getObjectSize(b);

            else return a.id - b.id;

        }

    }

And I've added a new function:

function getObjectSize(obj) {
        var box = new THREE.Box3().setFromObject(obj.object);
        var target = new THREE.Vector3( 0, 0, 0 );

        box.getSize(target);

        return target.length();
    }

@mrdoob is it even worth a Pull-Request?

[mrdoob]

@ranbuch Your approach would only fix your use case. It would break all other use cases (objects not in the same position).

However, combining your intention with the one in #13857... One approach would be to take the object's center and displace that point towards the camera using the boundingSphere radius.

[ranbuch]

It would break all other use cases (objects not in the same position).

I agree that it would work only in case an object is inside another bigger object but the current fallback behavior is sorting by id witch is, to the best of my knowledge, arbitrary so I would consider it an improvement (could be absolutely wrong here).

Any why, I have tried your approach, witch is the right one no doubt but again, not sure about my implementation.

It's working for my GLB and the case from #13857 jsfiddle here

Unfortunately, the solution involves adding a global variable (I'm not a fan) on line 327:

var CurrentCamera = null;

Assigning the camera on line 14,434:

CurrentCamera = camera;

Altering the reversePainterSortStable function on line 17,164:

function reversePainterSortStable( a, b ) {
        if ( a.renderOrder !== b.renderOrder ) {

            return a.renderOrder - b.renderOrder;

        } if (CurrentCamera && a.object && a.object.isMesh && b.object && b.object.isMesh ) {
            // In case the objects are of type Mesh we'll sort by mesh size
            // that why we might solve the transparency render order issue
            return getDistanceFromCamera(b, CurrentCamera) - getDistanceFromCamera(a, CurrentCamera);

        } else if ( a.z !== b.z ) {

            return b.z - a.z;

        }
        else return a.id - b.id;
    }

And adding a new function called getDistanceFromCamera:

function getDistanceFromCamera(object, camera) {
        var distance = new THREE.Vector3();
        distance.subVectors(camera.position, object.geometry.boundingSphere.center);
        return distance.length();
    }

@mrdoob what do you think?

[ranbuch]

On second thought, I'm taking the center of the geometry.boundingSphere when I'm supposed to take the nearest point to the camera?

Would it matter?

Can 2 Meshes overlap?

[mrdoob]

Can 2 Meshes overlap?

They sure can. But, unless we implement order-independent-transparency, we won't be able to produce the correct result.

[WestLangley]

@ranbuch Sorting by id is deterministic, and hence can help limit flickering. Your time is much better spent studying the literature. Learn about current approaches to handling transparency in modeling and in real-time rendering.

[ranbuch]

Sorting by id is deterministic, and hence can help limit flickering
👍

Learn about current approaches to handling transparency in modeling and in real-time rendering

Sure will.

@mrdoob so you think this solution won't work?

Should I just stop wasting your time and leave the rest to you guys?

[mrdoob]

Sorry, but the problem is way more complex than that.

Unfortunately transparency is still a unsolved problem in realtime computer graphics. In order to understand the problem you need to understand how GPUs draw things on the screen and what the z-buffer is.

This talk by @unconed may be a good start: https://acko.net/tv/webglmath/

[ranbuch]

While this source have taught me a lot I still was unable to see the problem with sorting the mesh the why that I did, but of course, that was just the tutorial witch didn't even concerned with the transparency issue.

After diving more into this issue I've learned that there's actually (as you've said) no perfect solution for this problem. not even by splitting overlap polygons . . .

I've gotta say, although I get the problem not having a solution still doesn't make sense to me as we don't encounter this issue in real life, hence, there must be a solution (probably beyond my reach).

Nevertheless, the fact of the matter is other Frameworks does handle this issue better then three-js (babylonjs, blend-4-web, windows 3D viewer (paint-3d), sketchfab etc..)

You may say, due-to the nature of the problem, that if they are solving this issue on a particular case and three-js is not there's probably other cases that three-js is doing a better job.

At the end of the day, though I get that sorting by id would prevent flickering (due-to it's consistency), that's just not the right why to sort! Even you are using it as a last resort (if there's no renderOrder and the z's are the identical).

For the time being, I'm not even sure if sorting by z is a better why then sorting by the distance from the camera . . .

And although I'm taking the center of the object, that's as good guess as taking the nearest or furthest vertex (again, due-to the nature of the problem).

Even if that would make three-js not better, but more consistent with the other realtime render engines that's a good thing as issues like that would just considered a modeling issue.

@mrdoob and the rest of the contributors, I'm truly sorry about the time that I've consumed due-to the lack of my understanding in 3D. I'm not expecting you to continue answering here anymore.

I'm considering using my solution as it seems to work better for me.
HEXA (my company) is creating hundreds (sometimes more) of 3D models every month and next week I'm hoping to start using three-js on our production.

I'll update here how my solution is working for me in the feature.

Thank you all.

[mrdoob]

Nevertheless, the fact of the matter is other Frameworks does handle this issue better then three-js (babylonjs, blend-4-web, windows 3D viewer (paint-3d), sketchfab etc..)

Any chance you can share screenshots of how these libraries/applications display your model? That's be super helpful!

Even if that would make three-js not better, but more consistent

Yeah, that's why I was wondering what logic all these other engines use.

[ranbuch]

Any chance you can share screenshots of how these libraries/applications display your model?

Sure, I'm on it

[looeee]

Nevertheless, the fact of the matter is other Frameworks does handle this issue better then three-js

I haven't tried the others, but I don't think Babylon handled it better. Sure, you can see the transparency, but as you rotate the bottle it jumps in and out in a big way. No transparency looks better than that, in my opinion.

[ranbuch]

babylonjs is indeed inconsistent.
Here's one angle that looks good:

And here's another that doesn't, you can see that the inside of the bottle is being rendered closer then the outside:

Next is Facebook 3D viewer. The word on the street is they are using three-js but no official confirmation. Well this image can almost confirm that they are:

Next is Microsoft with their mixed reality viewer (same as paint-3d to my knowledge):

They also have an inconsistency issue as from some angle you can see the inside of the bottle is being rendered closer then the bottle cap:

Next is gltf viewer android app:

and gltf viewer ios app:

Witch both looks the same. I had trouble changing the angle of the asset though.
I think those application are both based on 8thwall witch based on unity although I don't think unity support GLTF.
Again it seems that the inside mesh is being rendered closer then the outside mesh (you can see the inside mesh on the glenfiddich stickers witch is not transparent).

I'm still working on getting the original file that this glb has been exported from and then I'll be able to this model on some more viewers that doesn't support gltf.

The last one is three-js with the suggested sorting change:
link
witch works good.

It's also working for this example:
before change
after change

and this one as well:
before change
after change

Looks good but:

1. If you'll look closer on the last one you may detect a glitch - a very specific angel that's being rendered with a wrong order although the sorting doesn't actually changes, nor should it, it's a small sphere inside a big sphere . . .
2. There's a significant performance hit witch may (or may not) be improved.

If the author wishes to continue experimenting this direction I will do so.

Any why, obviously, if it's working on 3 examples it doesn't mean anything.

If anything I would suggest offering this approach as a secondary why to render scene in case the developer is explicitly asking for it in additional to the renderOrder one.

Something like WebGLRenderer.opacitySortApproach = THREE.SOME_ENUM

[mrdoob]

Next is Facebook 3D viewer. The word on the street is they are using three-js but no official confirmation.

Yo only need to open the developer console to see they are using three.js 😁

[mrdoob]

Actually. Because we use ids in the sort, you can work around this issue by changing the order of the objects in the scene in your authoring tool.

[ranbuch]

I'm not sure what do you mean by that. The sort might change according to the camera position, model position, animations . . .?

I'm sure I can improve the complexity at least somewhat by indexing the nested meshes and maybe re-index by somehow intercepting on scene changes?

I'm not that concerned with the performance at the time, I'm just not sure if the is right why.
First I would like to get to the right algorithm and then improve the frame rate.

Are there some other sources / examples (with transparency) you can direct me too so I can further test the sorting mechanism?

[mrdoob]

I'm not sure what do you mean by that.

What software did you use to create the model?

[mrdoob]

Are there some other sources / examples (with transparency) you can direct me too so I can further test the sorting mechanism?

Here's the most complex transparent situation I can think of:

spheres-transparent.glb.zip

[ranbuch]

What software did you use to create the model?

If I'm not mistaken it was done with maya, I'm not this model creator.

spheres-transparent.glb.zip

looks complecated enought for now 😉
I'll play with it soon enough.

[ranbuch]

BTW this is how it looks like with the current solution:
http://vqa.hexa3d.io/index.html?load=/models/spheres-transparent.glb

[ranbuch]

I know that's not a general solution but theoretically, If I'll join the 2 meshes together there wont be any sorting to do. That would solve the problem for scenes that can afford merging their meshes to a single mesh.

Tried something but it's terrible, I'll think of something else . . .

[mrdoob]

I know that's not a general solution but theoretically, If I'll join the 2 meshes together there wont be any sorting to do.

That's incorrect. These spheres are double sided, so the back of the spheres should be renderer first. GPUs do not sort triangles so that's why you see some stripes. Some triangles in the front get rendered before triangles on the back.

[mrdoob]

I realize the screenshot I attached is misleading. I didn't say that it's incorrect.
I've tried to load the glb in Blender and render it from there, but I'm not able to make the spheres transparent...

[ranbuch]

so that's why you see some stripes

I've actually wondered about that.

Looks like the only way is to create a pseudo mesh dynamically instead / in additional to the original meshes or to split meshes.

Can a solution like that ever be fest enough?

I'm not able to make the spheres transparent...

This is how the spheres looks like on 8thwal's android glft app:

I wonder how they are doing that . . .

[ranbuch]

@mrdoob

spheres-transparent.glb.zip

Put aside this model issues, I've improved my solution performance so no frames should be dropped.

If you'll look closer on the last one you may detect a glitch

Also, I've solved this glitch.

Keep in mind that for the sorting mechanize to work the WebGLRenderer constructor should be invoked like this:

THREE.WebGLRenderer({transparentSortLogic: THREE.CAMERA_PROXIMITY});

when transparentSortLogic's default value is THREE.Z_AXIS.

Do you want me to create a PR?

[mrdoob]

Something like renderer.setOpaqueSort( sortFunction ) and renderer.setTransparentSort( sortFunction ) may be a nicer API.

[ranbuch]

So you want the developer to write the sorting functions?
Is sortFunction a function or an Enum?

[ranbuch]

O.K. I've changed src/renderers/webgl/WebGLRenderLists.js and src/renderers/WebGLRenderer.js.
You can now call renderer.setOpaqueSort( sortFunction ) and renderer.setTransparentSort( sortFunction ) with an THREE.WebGLRenderer instance.

[pailhead]

I didn't get super crazy different results with different transparency sorting, but other objects intersecting this model do render differently:

https://raw.githack.com/pailhead/three.js/depth-peel-stencil-gltf/examples/webgl_materials_depthpeel.html

re:

Yeah, that's why I was wondering what logic all these other engines use.

I wonder how much of this is the responsibility of the "engine". There are probably many algorithms, and much different logic that can be applied to solve this problem (and other problems).

Rather than reverse engineering some other engine, i prefer to experiment with three.js. The "core" of three is perfect for this, the "engine" part is not.

I can hack away with shaders, the gl context but eventually you hit a wall.

For example, this approach to transparency uses the stencil buffer, without it i'm not sure if it could even be considered for some kind of production. Three has abstractions for depth ops but not for stencil ops. Render targets use their own stencil buffers and three gets in the way of sharing them, which is something that is possible with WebGL. I'm much more shocked by this than transparency (which is complex) failing if that makes sense :)

Other examples that fall in this category are normal maps, and shader chunks.

I'm not sure what exactly makes the "engine" in three's context. My guess is something that:

renders 3d and transparency just works

meaning that keeping the existing mesh.material.transparent = true api is great for that. But when it starts failing, my fear is that it's really hard to implement it in the "engine". Even the topic here refers to gltfs, even though this is a problem you can encounter with dynamically created scenes. This pattern can be seen in many issues here -> "gtlf doesn't work" when it's actually something that doesn't quite work with three itself.

On the other hand if i consider some kind of a "core" that is super flexible and extensible, one could do:

const scene = new THREE.Scene()
//...add much stuff to scene

const improvedTransparency = require('three-improved-transparency')(renderer)

improvedTransparency.render(scene, camera)

But the way three.js is built now this seems impossible to do. I don't know how other packages do it but i've seen things get refactored and split into different repositories and such.

Bottom line, it's much easier to pinpoint:

I can't assign my render buffer to a render target, it always creates its own and i don't have access to it (can be used for many effects)

Than

Gltf is not showing up "properly"

:)

[pailhead]

Here are the spheres in the bottle and outside the bottle: (need to hit "enable" twice in the demo to turn off the transparency sorting that happens first time by mistake)

[pailhead]

Still has some issues: