Arfoundation-samples: Modifications to native depth map images are undocumented

[sam598]

According to this comment https://github.com/Unity-Technologies/arfoundation-samples/issues/190#issuecomment-693542595 there may be undocumented changes being done to depth maps coming from native device APIs:

we do a fair amount of smoothing on the depth image already

It is not entirely clear what this comment means. If Unity is doing additional post processing on depth maps that can lead to an undesirable and unrecoverable loss in depth map quality.

I would like to see better documentation about which depth map settings correspond to unmodified images from the native API (e.g. sceneDepth and smoothedSceneDepth), which (if any) are being post-processed by Unity, and how they are being modified.

[tdmowrer]

The exact post-processing is subject to change and varies by platform, but one thing we could clarify in our docs is that the "fastest" mode always mean "no processing". So that will give you the unmodified image.

sceneDepth and smoothedSceneDepth

Note that the smoothing we do is a _per-frame_ smoothing of the sceneDepth while ARKit's smoothedSceneDepth is a _temporal_ smoothing. It is intended for static objects; if an object were to move, you would get a sort of "after image" effect.

[sam598]

one thing we could clarify in our docs is that the "fastest" mode always mean "no processing". So that will give you the unmodified image.

That would be great.

Note that the smooth we do is a _per-frame_ smoothing of the sceneDepth while ARKit's smoothedSceneDepth is a _temporal_ smoothing.

Understood. I would still recommend making different options from the native device accessible and clearly labelled, so that the consumer of the AR Foundation API can choose what is best for their application or use case (e.g. fast occlusion vs object scanning).

[rdtredinnick]

Following up on this - I am wondering if there are any plans to expose "smoothedSceneDepth" values directly through ARFoundation? Perhaps a flag on the session or something that would then mean TryAcquireEnvironmentDepthCpuImage and TryAcquireEnvironmentDepthConfidenceCpuImage returns smoothedSceneDepth values instead of sceneDepth.