Blenderproc: Truncated object detection

[SnowdenLee]

Hello,

I'm wondering if there is a way to detect if the chosen object (which has one special property to identify it) is truncated or heavily occluded in the image and if such situation happens then let the camera sampler to give up this pose and resample. I have tried to let the camera sampler to "look at" this chosen object, but then the object is always around the center of the image, which is not desired.

Do you have any suggestions or ideas? Thanks in advance.

[cornerfarmer]

Hey @SnowdenLee ,

you can take a look at the check_if_objects_visible parameter of the CameraSampler. This makes sure that your object is always visible in the image, otherwise the pose is discarded.
Internally this will send out raycasts from the camera and look if one of them hits the object.

At the moment this does not look at to which degree the object is visible (if one ray hits the object, thats enough). However, you could adjust that functionality, by changing the code here: https://github.com/DLR-RM/BlenderProc/blob/fe5ea258acb9e3ec11e0d4c6f380321e01c4e751/src/camera/CameraSampler.py#L479

You could for example count the number of rays that hit the object and only allow the camera pose if this number is big enough.

[SnowdenLee]

Hi,
thanks for the answer. Actually, I already tried this. But somehow it still can't guarantee that the object is not truncated. Sometimes the object is close to the camera, then many rays can hit the object, but in fact only part of the object is in the image. Also it's hard to choose a suitable threshold since sizes of objects are different.
So I'm wondering if there is a better way to deal with it, e.g., calculate the exact visible fraction of the object or sth like this.

[themasterlink]

Hey,

I think this ray is not checked for your desired clipping frustum. What you need to do is check if the ray hits inside of your desired clipping frustum, you can use its length to check that.

So I'm wondering if there is a better way to deal with it, e.g., calculate the exact visible fraction of the object or sth like this.

Yes, that would be an option, but without rendering this is quite hard, as you don't know which objects are covering which.
If you want to be a 100 % certain that it is correct, you could always render it with a sample amount of 1, use the instance semantic segmentation and then count the pixels. And then only use the camera poses which are good enough, of course you would need to save them and use them then in a later BlenderProc run.

Best,
Max

[cornerfarmer]

I don't think rendering the scene and checking the segmentation image will get you any further. check_if_objects_visible is actually an approximation of the result which you would get from rendering (rays are only sent through the frustum). By increasing sqrt_number_of_rays it gets closer and closer towards it.

However, in the end both ways can only calculate the fraction of your image that is covered by the object, but not the fraction of the object that is visible, which is what @SnowdenLee wants, as far as I understood it?

[SnowdenLee]

@cornerfarmer yes you understood it correctly. The occlusion can be somehow detected by both ways but not the truncation.

[cornerfarmer]

One way of making sure the whole object is in the image, would be the following:

1. Check that the object is not at the border of the image: Make sure that no ray at the border of the frustum hits the object.
2. Hide every other object in the scene via object.hide_viewport = True and do the ray check again. Then compare how many rays hit your object before and after hiding. If there is a big difference, then you have heavy occlusion and you can discard the camera pose.

You might need to increase sqrt_number_of_rays a bit, to increase accuracy.

Let me know if you need any further help :)

[SnowdenLee]

Thanks a lot. I will try this out :)

[MartinSmeyer]

If you also want the objects that touch the border you could alternatively render instance masks at both, the original image size and with a strongly increased frustrum (keep focal length constant, increase resolution) and then compute the ratio of pixels for your instance. Finally, render the RGB/depth images only for the camera poses that passed this test.

I guess this solution is a bit finer and faster than the check_if_objects_visible check, but of course not as nicely in place.

[themasterlink]

Thanks to all of you, I think we can close this issue for now :)