Pytorch3d: Does the volumetric function need cameras an input?

[jonilaserson]

The comments and tutorial state that the volumetric_function given to ImplicitRenderer need only have a RayBundle as a parameter. Yet the forward method of ImplicitRenderer forces the user to add a cameras argument as well (or to add **kwargs):

        rays_densities, rays_features = volumetric_function(
            ray_bundle=ray_bundle, cameras=cameras, **kwargs
        )

Is this intended?

[nikhilaravi]

@jonilaserson it depends what your volumetric_function does. In the tutorials, the volumetric_function only requires the ray_bundle but in other use cases you might want to transform the ray points using the cameras so it might be helpful to pass them in. In the forward method of the ImplicitRenderer the cameras are required so they are already available to be passed into the volumetric_function https://github.com/facebookresearch/pytorch3d/blob/4bb3fff52b7e26ec0f013021cb26fab7db3d8e0b/pytorch3d/renderer/implicit/renderer.py#L149-L151 If the volumetric_function doesn't require cameras then they will be ignored.

@davnov134 can explain further!

[davnov134]

To clarify, we included the cameras in the forward pass of the implicit function as well, because recent state-of-the-art methods often use camera-derived information (projections of ray points to images in IDR/IBR, ray directions, etc ..) to make the outputs of the implicit function (e.g. colors) dependent on camera viewpoints.

Indeed, as you mentioned, this requires users to add at least *kwargs to the impilcit function's forward pass. Here, we believe that the *kwargs requirement is ok since the goal of the API is to be as flexible as possible to facilitate custom modifications of the code.

[jonilaserson]

Makes sense. Probably good to document that ImplicitRenderer is also sending the cameras to the volumetric_function.
Because currently the docstring is wrong:
https://github.com/facebookresearch/pytorch3d/blob/4bb3fff52b7e26ec0f013021cb26fab7db3d8e0b/pytorch3d/renderer/implicit/renderer.py#L39-L42
If the user will follow this instruction, the code will crash. Also the given example will crash as well:
https://github.com/facebookresearch/pytorch3d/blob/4bb3fff52b7e26ec0f013021cb26fab7db3d8e0b/pytorch3d/renderer/implicit/renderer.py#L60-L66

[davnov134]

Great, thanks for this, we will update the docstring!