Open3d: Z-buffering in ColormapOptimization

[samarth-robo]

Is your feature request related to a problem? Please describe.
The colormap optimization library works great if the depth image is not noisy. Thank you for open-sourcing a very useful tool!
However, in my case, the depth maps are noisy, and not uniformly. For example, a sphere may be distorted at the sides so that it looks like an ellipsoid. So I need to loosen the depth_threshold_for_visibility_check, but that ends up spreading out the projected texture to some incorrect mesh faces.
So I was wondering if visibility can be determined by doing a Z-buffer test instead of comparing the rendered depth to sensor depth.
I'm happy to work on a PR if you think this is a good idea, and have some pointers!

Describe the solution you'd like
Determine vertex visibility in MakeVertexAndImageVisibility() using Z-buffering instead of comparing the rendered depth to sensor depth.

Describe alternatives you've considered

Additional context

[syncle]

Hi @samarth-robo. This is very good suggestion. We already have ingredients for this - rendering depth frames from mesh is here, and you can substitute depth image with rendered one and put new rgbd_images to the color_map_optimization. Could you contribute for this?

[samarth-robo]

Yes, working on it.

[samarth-robo]

Hi @syncle, please see this repository with my code. I showed a proof of concept there by adding random noise to the depth maps from the fountain dataset used in the tutorial.

• Currently I save the rendered depth maps to disk and read them, since I don't have headless rendering.
• I found that the Open3D renderer assumes the principal point is at the center of the image, and that the X and Y focal lengths are equal. So I had to apply an affine transform to the rendered depth-map, since my real-world camera doesn't follow these assumptions. This affine transform is mathematically guaranteed to 'simulate' the real-world camera, while rendering with the ideal camera.
• Let me know if you have ideas about how this can be integrated into the C++ code (if you want it to be)
• This method makes the texture mapping results on my own dataset much better, so once again I want to thank you guys for contributing this code!

[syncle]

This is great @samarth-robo! I didn't aware uneven focal length can make many hassles. I will open a new issue regarding uneven focal length and try to find the best way to merge this contribution in a proper way.

[germanros1987]

@syncle what would you like to do with this issue?

[zengletian1491]

Yes, working on it.

Is z-buffering updated in the newer-version Open3D (Open3D v0.11.2) now？Thank you.
@adamlofts @benjaminum