Filament: Depth Support

[HemanParbhakar]

Does Filament support object occlusion when combined with arcore?

[romainguy]

Yes you can render to the depth buffer or do the occlusion yourself in your materials.

[HemanParbhakar]

is this any sample available for that as i was using sceneform now google has suddenly stopped its support.

[vortice3D]

Hi @HemanParbhakar and @romainguy:

Regarding the artifacts reported here, here and here (between others issues), when trying to implement an occluding-not-shown geometry under ARCore-Filamet-Sceneform that, as far as I understand, is due to the need of changing the rendering priority, being the camera feed rendered first (priority 0) instead to last (priority 7) as is by default, I wonder what type of approach is being followed in the new Depth API, in order to avoid that type of problems.

Best regards.

[HemanParbhakar]

so @vortice3D filament supports depth ? If yes, then can you please provide a proper documentation.

[romainguy]

Yes, Filament supports writing the depth buffer only. You can do this in your materials by disabling color writes. See the docs for materials: https://google.github.io/filament/Materials.html

[HemanParbhakar]

@romainguy but how will depth buffer works as it was only works in open gl and ndk for arcore.

[vortice3D]

HI @HemanParbhakar:

The way to implement an "occluder" material is:

1) We start with a general occlusive material definition (.mat file):

material {
    name : "Occlusion material",
    shadingModel : unlit,
    colorWrite : false,
    depthWrite : true
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor = vec4(0.0);
    }
}

---

2) Then, we must compile that material using the Filament “matc” tool, this way:
matc -o ./materials/bin/YOUR_MATERIAL_NAME.matc ./materials/src/YOUR_MATERIAL_NAME.mat

Warning! Please note that the material mus be compiled with each new version of Filament.

---

3) Also, you need to change the rendering priority, being the camera feed rendererd first (priority 0) insted to last (priority 7) as is by default. The priority must be: Camera > Occluder renderables > Others renderables

Anyway, people are reporting visual artifacts here and there following this workflow. That is the reason I asked @romainguy for the different way that, obviously (as artifacts are not shown), the Depth API functionality is following for its "occluders".

Best regards.

[HemanParbhakar]

@vortice3D Thanks for your time and comment.
Actually there is lot of confusion I just wanna add the AR objects in the view add occlusion with them.
The Priority is not avaible in Sceneform1.17.1
arFragment.getArSceneView().setCameraStreamRenderPriority(Renderable.RENDER_PRIORITY_FIRST);
So I am using RenderableManager.build().setPriority(0);
But how to set it different for each Renderable like if I have 10 AR Objects.
Also once I have the matc how do I set that upon tap I have to use viewRenderable (a layout file) and some cubes with that matc file.

[zirman]

Here is an example project on getting object ARCore occlusion working with Filament.
https://github.com/zirman/arcore-filament-example-app

[vortice3D]

@vortice3D Thanks for your time and comment.
Actually there is lot of confusion I just wanna add the AR objects in the view add occlusion with them.
The Priority is not avaible in Sceneform1.17.1
arFragment.getArSceneView().setCameraStreamRenderPriority(Renderable.RENDER_PRIORITY_FIRST);
So I am using RenderableManager.build().setPriority(0);
But how to set it different for each Renderable like if I have 10 AR Objects.
Also once I have the matc how do I set that upon tap I have to use viewRenderable (a layout file) and some cubes with that matc file.

Hi @HemanParbhakar:

Well as you say, sadly, the render priority API is disappeared under Sceneform1.17.1. This situation, along with the reported artifacts shown when using such a technique (of course, using previous Sceneform versions which did let you use render priority) was the point where I've just abandoned this trail.

Then DepthAPI came life and I thought: "Hey this guys/girls are implementing occlusion once the depth info is obtained. How the render-thing of it will be working?.

So I asked here, and well, maybe the answer are the vertex shaders from @zirman. But I presume is not an easy-to-translate-to-sceneform solution. Well, not have Sceneform is the not-sunny-side-of-the-street, at least not as sunny as iOS' is, which keeps supporting SceneKit (counterpart of the deceased Sceneform, God rest his soul!).

Best regards.

[romainguy]

You can implement support for ARCore's depth API in several ways. The "easiest" one that will give decent results is to sample the depth texture from the object's material and discard fragments manually. By sampling several points in the depth map you can even feather the occlusion to avoid aliasing effects. Another solution is to write to the depth buffer using a full screen triangle and modifying the fragment's Z value from the fragment shader. You'll lose early-Z but that's not a big deal for this kind of scenes.

[zirman]

@romainguy Is there a way to explicitly specify the value written to the depth buffer in the pixel shader?

[romainguy]

You can write to gl_FragDepth to achieve this.

[zirman]

Is that available from Filament? https://google.github.io/filament/Materials.html

[romainguy]

Yes, it should work just fine. We haven't tested it across platforms which is why we don't document it but it'll work on Android.

[zirman]

Would you kindly provide an example? :-)

[romainguy]

Of writing depth? Just do gl_FragDepth = value; // value is between 0 and 1. So with the ARCore depth map, you'd probably just assign texture(arcore_depth_map, getUV0()) to gl_FragDepth.

[zirman]

Nice. I didn't know that variable was there. I thought it would be a field on material. Thanks!

[HemanParbhakar]

@romainguy @zirman the easiest approach would be to use the Arcore Depth API ? For filament the person first needs to understand how the graphics would and each term which is a long task

[romainguy]

Materials are turned into GLSL shaders, so you technically have access to everything a GLSL shader has access to (gl_FragCoord, etc.). But like I said above, unless we can guarantee it will work on all target platforms/APIs (metal, vulkan, etc.) we don't recommend using those things. But if you are aware of this and you're ok with it… :)

[HemanParbhakar]

Of writing depth? Just do gl_FragDepth = value; // value is between 0 and 1. So with the ARCore depth map, you'd probably just assign texture(arcore_depth_map, getUV0()) to gl_FragDepth.

https://developers.google.com/ar/develop/java/depth/developer-guide#retrieve-depth-maps
This is function it returns DepthImage

[HemanParbhakar]

@vortice3D Thanks for your time and comment.
Actually there is lot of confusion I just wanna add the AR objects in the view add occlusion with them.
The Priority is not avaible in Sceneform1.17.1
arFragment.getArSceneView().setCameraStreamRenderPriority(Renderable.RENDER_PRIORITY_FIRST);
So I am using RenderableManager.build().setPriority(0);
But how to set it different for each Renderable like if I have 10 AR Objects.
Also once I have the matc how do I set that upon tap I have to use viewRenderable (a layout file) and some cubes with that matc file.

Hi @HemanParbhakar:

Well as you say, sadly, the render priority API is disappeared under Sceneform1.17.1. This situation, along with the reported artifacts shown when using such a technique (of course, using previous Sceneform versions which did let you use render priority) was the point where I've just abandoned this trail.

Then DepthAPI came life and I thought: "Hey this guys/girls are implementing occlusion once the depth info is obtained. How the render-thing of it will be working?.

So I asked here, and well, maybe the answer are the vertex shaders from @zirman. But I presume is not an easy-to-translate-to-sceneform solution. Well, not have Sceneform is the not-sunny-side-of-the-street, at least not as sunny as iOS' is, which keeps supporting SceneKit (counterpart of the deceased Sceneform, God rest his soul!).

Best regards.

I totally agree. Abandoning Sceneform have make many projects to halt. Due to which developer have to get in depth to understand OpenGL Stuff. Its a nightmare.

[HemanParbhakar]

@vortice3D were you able to achieve occlusion in physical world?

[vortice3D]

@vortice3D were you able to achieve occlusion in physical world?

Hi @HemanParbhakar:

I've just read your new issue on it and, with the fresh info, I'm going to re-think my own approach to all this.

As soon as I can reach any conclusion I'll let you know.

Best regards.

[zirman]

This shader works without needing to tessellate the camera texture. Instead the depth is set in the pixel shader.

material {
    name : depth,
    shadingModel : unlit,
    blending : opaque,
    vertexDomain : device,
    parameters : [
        {
            type : samplerExternal,
            name : cameraTexture
        },
        {
            type : sampler2d,
            name : depthTexture
        },
        {
            type : float4x4,
            name : uvTransform
        }
    ],
    requires : [
        uv0
    ]
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor.rgb = inverseTonemapSRGB(texture(materialParams_cameraTexture, getUV0()).rgb);
        vec2 packed_depth = texture(materialParams_depthTexture, getUV0()).xy;
        float depth_mm = dot(packed_depth, vec2(255.f, 256.f * 255.f));
        vec4 view = mulMat4x4Float3(getClipFromViewMatrix(), vec3(0.f, 0.f, -depth_mm / 1000.f));
        float ndc_depth = view.z / view.w;
        gl_FragDepth = 1.f - ((ndc_depth + 1.f) / 2.f);
    }
}

vertex {
    void materialVertex(inout MaterialVertexInputs material) {
        material.uv0 = mulMat4x4Float3(materialParams.uvTransform, vec3(material.uv0.x, material.uv0.y, 0.f)).xy;
    }
}

[vortice3D]

@vortice3D were you able to achieve occlusion in physical world?

Hi again @HemanParbhakar:

First, I must remark that my case of use for all this occlusion thing, is not related to a depth capable device providing z-depth info about real elements that are occluders of virtual occluded elements.

In my implementation I have a pre-created 3D scene with occluders and occluded meshes, that I calibrate in a previous step, in order to make match the occluders ones with their physical counterparts.

My problems here come mainly with the way to set the different render priorities.

---

This said, after reading in detail your issue #3162, and also the #2423 (that I had missed at the time), I understand that:

A) for the renderables (occluder and occluded geometries), all the magic related to the render priorities is done through the following kind of filament-code snippet (note that I'm only showing lines of code related to this subject):

...
myRenderable = EntityManager.get().create();
RenderableManager.Builder(1)
    ...
    .priority(5)
    .material(0, materialInstance)
    .build(engine, myRenderable);
scene.addEntity(myRenderable);
...

B) about the camera render priority, as you explain a modification of the ARSceneView object must be done on the (unofficial) 1.17 Sceneform version (last line cameraStream.setRenderPriority(0);):

private void initializeCameraStream() {
    cameraTextureId = GLHelper.createCameraTexture();
    Renderer renderer = Preconditions.checkNotNull(getRenderer());
    cameraStream = new CameraStream(cameraTextureId, renderer);
    cameraStream.setRenderPriority(0);
}

---

Back to the needs of my development, I want to iterate through the whole 3D asset scene, applying render priorities to the meshes depending on they are occluders or occluded elements. When they are occluders, I need also to apply to them the occluder material.

My code looks something like this:

protected void onCreate(Bundle savedInstanceState) {
   ...
   arFragment.setOnTapArPlaneListener(
      (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
         if (renderable == null) {
             return;
         }
         // Create the Anchor
         Anchor anchor = hitResult.createAnchor();
         // Create the Anchor Node
         AnchorNode nodeAnchor= new AnchorNode(anchor);
        nodeAnchor.setParent(arFragment.getArSceneView().getScene());
        // Create the transformable model and add it to the anchor.
        TransformableNode nodeModel = new TransformableNode(arFragment.getTransformationSystem());
        nodeModel.setParent(anchorNode);
        nodeModel.setRenderable(renderable);
        nodeModel.select();
        //
        doAllThatFilamentStuff(nodeModel);
    });
    ...
}
....
//
private void doAllThatFilamentStuff(Node node){
    FilamentAsset filasset = node.getRenderableInstance().getFilamentAsset();
    if (filasset!=null && filasset.getAnimator().getAnimationCount() > 0) {
        animators.add(new AnimationInstance(filasset.getAnimator(), 0, System.nanoTime()));
    }
    //
    Engine engine=EngineInstance.getEngine().getFilamentEngine();
    if(engine!=null){
        RenderableManager rm=engine.getRenderableManager();
        if(rm!=null){
            for (int entity: filasset.getEntities()) {
                if(rm.getInstance(entity)==0){
                    continue;
                }

                //if the geometry is the occluder one assign the occluder material to it
                if(filasset.getName(entity).equals("Occluder_geo")){
                    try{
                        InputStream inStream = this.getResources().openRawResource(R.raw.sceneform_occluder_material);
                        byte[] filamat = new byte[inStream.available()];
                        inStream.read(filamat);
                        inStream.close();
                        //material data buffer
                        ByteBuffer buffer=ByteBuffer.wrap(filamat);
                        //material builder
                        com.google.android.filament.Material.Builder myBuilder=new com.google.android.filament.Material.Builder();
                        //material Filament
                        Material occluderMaterial=myBuilder.payload(buffer,buffer.remaining()).build(engine);
                        occluderMaterial.setDefaultParameter("color", 1.0f,0.0f,0.0f);  //debug
                        rm.setMaterialInstanceAt(rm.getInstance(entity),0,occluderMaterial.createInstance());
                     }
                     catch (IOException e){
                         e.printStackTrace();
                     }
                }
                ...
            }
        }
    }
}

---

As a colophon, as I'm using the classic Sceneform code for loading GLTF/GLB files, based on the use of ModelRenderable.builder(), I can't have access to the benefit of setting any priority, as RenderableManager.Builder let to do.

And that's the cause I haven't been able to implement successfully occluding at the moment. Maybe I must resign to use ModelRenderable.builder(), using RenderableManager.Builder instead. In that approach I think I'll need to use separated GLTF/GLB files for occluders and occluded meshes.

Best regards.

[HemanParbhakar]

This shader works without needing to tessellate the camera texture. Instead the depth is set in the pixel shader.

material {
    name : depth,
    shadingModel : unlit,
    blending : opaque,
    vertexDomain : device,
    parameters : [
        {
            type : samplerExternal,
            name : cameraTexture
        },
        {
            type : sampler2d,
            name : depthTexture
        },
        {
            type : float4x4,
            name : uvTransform
        }
    ],
    requires : [
        uv0
    ]
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor.rgb = inverseTonemapSRGB(texture(materialParams_cameraTexture, getUV0()).rgb);
        vec2 packed_depth = texture(materialParams_depthTexture, getUV0()).xy;
        float depth_mm = dot(packed_depth, vec2(255.f, 256.f * 255.f));
        vec4 view = mulMat4x4Float3(getClipFromViewMatrix(), vec3(0.f, 0.f, -depth_mm / 1000.f));
        float ndc_depth = view.z / view.w;
        gl_FragDepth = 1.f - ((ndc_depth + 1.f) / 2.f);
    }
}

vertex {
    void materialVertex(inout MaterialVertexInputs material) {
        material.uv0 = mulMat4x4Float3(materialParams.uvTransform, vec3(material.uv0.x, material.uv0.y, 0.f)).xy;
    }
}

@zirman so this will work to do occlusion in physical environment at real time.

[zirman]

@HemanParbhakar Yes. It does what the last pixel shader did but instead of having a grid for the camera texture where the depth is modified in the vertex shader, this just modifies the depth value in the pixel shader. It doesn't work on backends that are not OpenGL unfortunately.

[HemanParbhakar]

@zirman so I Create this filamat file and assign that to model and then the occlusion with real environment will work ?

[zirman]

@HemanParbhakar Look at this example project to see how to use it. That has every detail you need to get it working. https://github.com/zirman/arcore-filament-example-app

[HemanParbhakar]

But I need It work on sceneform.

[steven-gao]

@vortice3D Thanks for your time and comment.
Actually there is lot of confusion I just wanna add the AR objects in the view add occlusion with them.
The Priority is not avaible in Sceneform1.17.1
arFragment.getArSceneView().setCameraStreamRenderPriority(Renderable.RENDER_PRIORITY_FIRST);
So I am using RenderableManager.build().setPriority(0);
But how to set it different for each Renderable like if I have 10 AR Objects.
Also once I have the matc how do I set that upon tap I have to use viewRenderable (a layout file) and some cubes with that matc file.

Hi @HemanParbhakar:
Well as you say, sadly, the render priority API is disappeared under Sceneform1.17.1. This situation, along with the reported artifacts shown when using such a technique (of course, using previous Sceneform versions which did let you use render priority) was the point where I've just abandoned this trail.
Then DepthAPI came life and I thought: "Hey this guys/girls are implementing occlusion once the depth info is obtained. How the render-thing of it will be working?.
So I asked here, and well, maybe the answer are the vertex shaders from @zirman. But I presume is not an easy-to-translate-to-sceneform solution. Well, not have Sceneform is the not-sunny-side-of-the-street, at least not as sunny as iOS' is, which keeps supporting SceneKit (counterpart of the deceased Sceneform, God rest his soul!).
Best regards.

I totally agree. Abandoning Sceneform have make many projects to halt. Due to which developer have to get in depth to understand OpenGL Stuff. Its a nightmare.

I totally agree with your point of view, my previous project also used sceneform. It is a headache to do migration now

[steven-gao]

But I need It work on sceneform.

do you implement depth on sceneform?