Pytorch3d: Error in Batched Rendering with PhongShader or GouraudShader

[shubhtuls]

I run into errors when trying to use batched rendering in case of meshes with a per-vertex texture, using both SoftPhongShader or SoftGouraudShader.

To reproduce the error, one can use the mesh rendering notebook and modify the last few lines in the 'Create a Renderer' cell to the following:

from pytorch3d.renderer import (
    SoftPhongShader,
    SoftGouraudShader
)

# Ignore texture of loaded mesh and create a mesh with per-vertex texture
verts = mesh.verts_list()
faces = mesh.faces_list()
verts_rgb = torch.ones_like(verts[0][None,:,:])
mesh = Meshes(verts, faces, textures=Textures(verts_rgb=verts_rgb))

# Create a renderer by composing a rasterizer and a shader.
renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, 
        raster_settings=raster_settings
    ),
    # both 'SoftPhongShader' and 'SoftGouraudShader' fail, but with different errors
    #shader=SoftPhongShader(
    shader=SoftGouraudShader(
        device=device, 
        cameras=cameras,
        lights=lights
    )
)

All the examples except the batched rendering run as expected. However, the batched rendering produces issues for either SoftPhong or SoftGouraud shaders (I think other shaders might also have similar issues).

For the Phong one, the error log is:

~/code/pytorch3d/pytorch3d/renderer/mesh/shader.py in forward(self, fragments, meshes, **kwargs)
    111 
    112     def forward(self, fragments, meshes, **kwargs) -> torch.Tensor:
--> 113         texels = interpolate_vertex_colors(fragments, meshes)
    114         cameras = kwargs.get("cameras", self.cameras)
    115         lights = kwargs.get("lights", self.lights)

~/code/pytorch3d/pytorch3d/renderer/mesh/texturing.py in interpolate_vertex_colors(fragments, meshes)
    109         fragments.pix_to_face.
    110     """
--> 111     vertex_textures = meshes.textures.verts_rgb_padded().view(-1, 3)  # (V, C)
    112     vertex_textures = vertex_textures[meshes.verts_padded_to_packed_idx(), :]
    113     faces_packed = meshes.faces_packed()

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

For the Gouraud one, the error log is:

~/code/pytorch3d/pytorch3d/renderer/mesh/shading.py in gouraud_shading(meshes, fragments, lights, cameras, materials)
    118     if len(meshes) > 1:
    119         lights = lights.clone().gather_props(vert_to_mesh_idx)
--> 120         cameras = cameras.clone().gather_props(vert_to_mesh_idx)
    121         materials = materials.clone().gather_props(vert_to_mesh_idx)
    122 

~/code/pytorch3d/pytorch3d/renderer/utils.py in gather_props(self, batch_idx)
    236                         msg = "batch_idx cannot have more dimensions than %s. "
    237                         msg += "got shape %r and %s has shape %r"
--> 238                         raise ValueError(msg % (k, idx_dims, k, tensor_dims))
    239                     if idx_dims != tensor_dims:
    240                         # To use torch.gather the index tensor (batch_idx) has

ValueError: batch_idx cannot have more dimensions than T. got shape torch.Size([58600, 3, 3]) and T has shape torch.Size([20, 3])

[shubhtuls]

I think the latter issue occurs because 'batch_idx' is being overwritten in the inner loop in these lines.

[nikhilaravi]

@shubhtuls thanks for the detailed explanation of the issue! I will try to reproduce the error as you have described and get back to you.

[nikhilaravi]

@shubhtuls I managed to reproduce the errors - thanks a lot for sharing such detailed information, it made it much easier to debug! There are three small fixes so you could add them in to your local clone if you're blocked and I can merge in the fix ASAP.

For the SoftPhongShader case, the error is because of extending the textures in the call to mesh.extend(batch_size). To fix this just change the view in pytorch3d/pytorch3d/renderer/mesh/texturing.py line 111 to reshape

https://github.com/facebookresearch/pytorch3d/blob/ba11c0b59cb53d50dc5f50e0e0148b3f2e43f39f/pytorch3d/renderer/mesh/texturing.py#L111

For the SoftGouraudShader case, you are right that the batch_idx is being overwritten. The gather_props function reshapes all the attributes of the class, so each iteration of the for loop should start with the same initial batch_idx which is passed into the function. A simple fix for this is to rename the input argument to batch_idx_init here https://github.com/facebookresearch/pytorch3d/blob/ba11c0b59cb53d50dc5f50e0e0148b3f2e43f39f/pytorch3d/renderer/utils.py#L180

Then add batch_idx = batch_idx_init.clone() before line 233 of pytorch3d/pytorch3d/renderer/utils.py here https://github.com/facebookresearch/pytorch3d/blob/ba11c0b59cb53d50dc5f50e0e0148b3f2e43f39f/pytorch3d/renderer/utils.py#L233

Let me know if you have any more problems!

[shubhtuls]

@nikhilaravi - thanks a lot for looking into this and suggesting the changes! I did unblock on my end by updating the local clone, so there is no urgency - please feel free to close the issue/merge to main at your convenience. Thanks!

[raincrash]

Hi, may I work on this good-first-issue PR @nikhilaravi or is someone working on fixing this already?

[nikhilaravi]

@raincrash I already have a fix for this issue. But have a look at the comments in other open issues if you'e interested in contributing :) Some of them have explanations for the necessary fix.

[nikhilaravi]

This should now be fixed by https://github.com/facebookresearch/pytorch3d/commit/5d3cc3569a03f44647857243efd0d80588a6785b.