Blenderproc: How to know if the object is in the camera view?

[b-yogesh]

I am trying to render a robot in a SunCG room, I have been successful at that. But some times, the images rendered do not contain the robot, so how can I know if the robot is visible from the camera before rendering? Is there any example of how to do that?
Thanks

[themasterlink]

There are several options, first you could sample the camera around the robot, like in this example where we sampled around an object from ShapeNet:

Here both time the location and the rotation depend on the result of the Entity.Getter:

https://github.com/DLR-RM/BlenderProc/blob/dbfe70e44410918063c4df1f00b700e6b20b93da/examples/shapenet_with_scenenet/config.yaml#L91-L125

Alternatively you could use the scene score mechanism for increasing the chance that cameras are accepted if the robot is inside the frame, which will make it more arbitrary.

https://github.com/DLR-RM/BlenderProc/blob/dbfe70e44410918063c4df1f00b700e6b20b93da/src/camera/CameraSampler.py#L100-L104

Here again select the robot with a Entity.Getter and then use a really high score like 200, to make sure that those scenes are rated higher.

Does this answer your question?

[themasterlink]

Lastly, to make sure that the robot is always in the frame you could add a function similar to this one, where we check the scene coverage:

https://github.com/DLR-RM/BlenderProc/blob/dbfe70e44410918063c4df1f00b700e6b20b93da/src/camera/CameraSampler.py#L431-L436

Here you could check that the robot is always on of the hit_objects if not the camera pose is rejected.

[ideas-man]

@b-yogesh As an addition to the @themasterlink first post.
getter.Attribute allows to get the location of the specific object in the scene:

        {
          "provider": "getter.Attribute",
          "entities": {
            "provider": "getter.Entity",
            "conditions": {
              "name": "MyObject"
            }
          },
          "get": "location"
        }

which can be passed as a value, for example, of the look_at parameter of the CameraSampler module.
getter.AttributeMerger, in its turn, can be used to introduce a random element to that location as a part of the sum (or average), if needed:

        {
          "provider": "getter.AttributeMerger",
          "elements": [
          {
            "provider": "getter.Attribute",
            "entities": {
              "provider": "getter.Entity",
              "conditions": {
                "name": "MyObject"
              }
            },
          "get": "location"
          {
            "provider": "sampler.Uniform3d",
            "min": [0, 0, 0],
            "max": [1, 1, 1]
          }
          ],
          "transform_by": "sum"
        }

which can be used in the same manner as the aforementioned one.

In case of getter.Entity being configured such that it returns only one object, there will be no difference between using getter.Attribute/getter.AttributeMerger and getter.POI as described in our posts, but will be when you try to select groups of objects (given you do not modify the result via summation or averaging with some other arbitrary vector).

P.S.: refer to the documentation of the the mentioned providers for more details.

[themasterlink]

Thanks to @ideas-man this should be solved.

[b-yogesh]

I am sorry for replying late to this issue, I was trying out those options and they worked out really well (especially the scene coverage). Thank you @themasterlink and @ideas-man for your prompt inputs. :)