Azure-kinect-sensor-sdk: Depth are interpolated at surface edge

[YuHuangjie]

Describe the bug

I notice that depth values at surface edges (depth discontinuities) are interpolated. This introduces wrong depth values and causes adhesion to another object. Examples are shown below.

Expected behavior

I wonder if this is the correct behavior of the camera SDK. Is there an approach for getting non-interpolated depth maps?

Screenshots

I put a hand in front of me, leaving a 10 cm space. This depth map is undistorted following the given sample. I didn't use any OpenCV Linear interpolation.

A closeup of my thumb is shown below. Clearly, depth values are interpolated along the edge. Note that I zoom in the image using 'nearest' method.

Desktop:

• OS with Version: Windows 1803
• SDK Version: 1.2.0
• Firmware version: 1.6.102075014

[SergiiPatar]

I have exactly the same issue. I compiled and tested the latest SDK (1.3.0-beta.0) and the problems remains.
I would be very nice to have a functionality to switch off the edge smoothing/interpolation

[rabbitdaxi]

@YuHuangjie Thanks for the feedback. Based on your description, you are talking about the undistortion example, is that correct? The example is code samples to show the many ways of how to undistort the depth image. Depends on situation,

• you can choose to use nearest neighbor if you want absolutely no interpolation at all.
• Or if you choose to use the bi-linear to achieve smoothness, agree that the depth discontinuity or the pixel between invalid/valid depth should be checked before interpolate. Please check this commit we added some threshold for the depth continuity check. If this cannot satisfy your purpose, please use the NN instead or propose a threshold for the depth discontinuity check.

[rabbitdaxi]

@SergiiPatar is your issue also related to the undistortion example? or is your issue about the depth image straight from the SDK?

[YuHuangjie]

@rabbitdaxi Thank you for your reply. The issue is not about undistortion. I tried removing undistortion completely and the problem still exists. So I think the issue is related to the depth map straight from the SDK.

[rabbitdaxi]

@YuHuangjie If this is the depth map from the camera itself, this can be explained by the principle of time of flight and the depth algorithm. Two points:

• Signal can be received at the edge pixel mixing both foreground and background, for these pixels we often invalidate them, but it is possible when foreground and background is close, they were not invalidated and result in a pixel may have less accurate depth.
• We do have filter in the depth engine, it is designed to keep depth discontinuity. But is also possible an explanation of what you see above.

[mbleyer]

It might be an artefact of the filter. If two objects have similar depths, the filter might blur the depth discontinuity. Can you confirm that this blurring gets reduced when you move your hand further away from your body?

[bearpaw]

Hi, I got the similar problem. From the visualization of the pointcloud, we can clearly see that the edge of the box is "blurred" with the desk surface as a "long tail".

I guess it is caused by the interpolation/smoothness filtering within the depthengine? It would be very helpful if we can control the smoothness, as been done for the body tracking.

[YuHuangjie]

@mbleyer If I move my hand further away, the blurring is reduced. In fact, boundary pixels are invalidated (black) in that case.

[catid]

Is this issue present in both K4A_DEPTH_MODE_NFOV_2X2BINNED and K4A_DEPTH_MODE_NFOV_UNBINNED?

[bearpaw]

Is this issue present in both K4A_DEPTH_MODE_NFOV_2X2BINNED and K4A_DEPTH_MODE_NFOV_UNBINNED?

It happens to me for all four modes.

[mbleyer]

Thanks for running the experiment. This seems to be an expected artefact of the smoothing filter used in the depth estimation software. Please note that changing the filter parameters is risky and might lead to very large depth errors (so-called dealiasing errors). We are therefore currently not planning to expose the filter parameters. I would recommend using a median filter on the depth map if these intermediate depth pixels represent a major issue.

[rabbitdaxi]

This issue is expected with the current version of hardware and the software along with it. The filter is needed as a core part of the raw to depth process which already have parameters to help preserving the edges when there is large depth discontinuity. User can based on the specific scenario and environment data to apply further filtering either through customized thresholds or incorporate color image information. At this point, we close this issue as by design. You can also provide feedback on this website for future product consideration.

[yipingp]

Hi, I got the similar problem. From the visualization of the pointcloud, we can clearly see that the edge of the box is "blurred" with the desk surface as a "long tail".

I guess it is caused by the interpolation/smoothness filtering within the depthengine? It would be very helpful if we can control the smoothness, as been done for the body tracking.

Hi, did you work it out or is there anything update? Thanks a lot!

[bearpaw]

@yipingp unfortunately there is no luck on my side. Please let me know if you can figure something out. Thanks :)

[yipingp]

@bearpaw Yes, sure

[mika-fischer]

@yipingp @bearpaw Did you find a good workaround for this issue. I also experience it:

This is a disc hovering over a flat surface. All the interpolated depth values are causing me a lot of headaches in segmentation...