Pytorch3d: Unexpected behavior in batched texture input

[aluo-x]

Using the code provided in pytorch3d examples to illustrate a point:

import pytorch3d
import torch
import numpy as np
from pytorch3d.io import load_objs_as_meshes
from pytorch3d.renderer import PointLights, look_at_view_transform, OpenGLPerspectiveCameras, RasterizationSettings, MeshRenderer, MeshRasterizer, SoftPhongShader, SoftSilhouetteShader, TexturesVertex
from pytorch3d.loss import mesh_edge_loss, mesh_normal_consistency, mesh_laplacian_smoothing    
!wget https://raw.githubusercontent.com/facebookresearch/pytorch3d/master/docs/tutorials/utils/plot_image_grid.py
from plot_image_grid import image_grid
import matplotlib.pyplot as plt
from tqdm.notebook import tqdm

batch_size = 20
cur_device = "cuda:0"
template_mesh = load_objs_as_meshes(["./ico_sphere_lvl4.obj"], device=cur_device)
meshes = template_mesh.extend(batch_size)

# Not correct here?
sphere_verts_rgb = torch.rand([1, meshes.verts_packed().shape[0], 3], device=cur_device, requires_grad=True)
meshes.textures = TexturesVertex(verts_features=sphere_verts_rgb) 
print(TexturesVertex(verts_features=sphere_verts_rgb)._verts_features_padded.shape)
print(len(meshes.textures.verts_features_list()))


# Get a batch of viewing angles. 
elev = torch.linspace(0, 180, batch_size)*0.0
azim = torch.linspace(-180, 180, batch_size)*0.0

R, T = look_at_view_transform(dist=2.7, elev=elev, azim=azim)
cameras = OpenGLPerspectiveCameras(device=cur_device, R=R, T=T)
lights = PointLights(ambient_color=((0.5, 0.5, 0.5),), diffuse_color=((0.3, 0.3, 0.3), ), specular_color=((0.2, 0.2, 0.2), ), location=[[0.0, 0.0, -4.0]], device=cur_device)
lights.location = torch.tensor([[0.0, 0.0, -3.0]], device=cur_device)
images = renderer(meshes, cameras=cameras, lights=lights).detach()

image_grid(images.cpu().numpy(), rows=5, cols=4, rgb=True)

After repeating a mesh 20 times, I set the texture of TexturesVertex class to have shape [1, 51240, 3]. I render the sphere from the same view point.

Expected behavior:

1. Rendering fails
2. Or - rendering succeeds but each sphere looks the same (since same view)

Actual behavior:

1. Each sphere has a different texture

Additional testing:

1. setting the texture to have shape [20, 2562, 3] works as expected

This seems to be different than the TexturesVertex documentation. Batch dimension of the texture when give as a tensor, should match len(meshes.verts_list()).

[gkioxari]

sphere_verts_rgb = torch.rand([1, meshes.verts_packed().shape[0], 3], device=cur_device, requires_grad=True)

This seems problematic! So you have a meshes data structure with 20 elements (aka len(meshes) = 20). But you are setting the textures to have batch size of one and the number of vertices to be equal to the packed shape, meaning the sum of all number of vertices in meshes. That is not the right way to use textures. The right way would be as follows

• If you have homogeneous meshes (yours are because you have used .extend) then you set sphere_verts_rgb = torch.rand_like(meshes.verts_padded(), device=cur_device, requires_grad=True) which sets textures to be a tensor of shape BxVx3 where B=20, V is the number of vertices in each mesh element in meshes (since meshes are homogeneous that is the same across all elements)

• If meshes is heterogeneous, then you initialize the textures with a list sphere_verts_rgb = [torch.rand_like(v) for v in meshes.verts_list()] and textures = TexturesVertex(sphere_verts_rgb)

This is also described here

https://github.com/facebookresearch/pytorch3d/blob/fc7a4cacc36ecf2b8feb996bdfb1203dee5d836b/pytorch3d/renderer/mesh/textures.py#L1214-L1226

and it follows the convention across the library. If you have homogeneous data structures, you construct with a tensor of shape BxVxC if you have heterogeneous data you construct with a list of VixC for i=0, .., B-1

[aluo-x]

I agree. I think it would be better for pytorch3D to raise an error. Since what I wrote should not be allowed (going by either the documentation or the docstring).

[gkioxari]

Absolutely!! We should! I think we have a check when users construct meshes = Meshes(verts=verts, faces=faces, textures=textures) but we should also make sure it checks when it is set as an attribute externally.

[gkioxari]

I marked it as enhancement and will leave it open until we fix it! Unless, you want to add a PR which I am happy to review!

[aluo-x]

I can potentially create a PR.

My only concern is how rendering succeeded, and how when viewed from the same camera we have objects with different textures. I suspect there is a reshaping somewhere.

Will take a look in a bit, wanted to confirm this behavior was not desired first.

[gkioxari]

It succeeds because during rendering we actually use the packed representation of the textures. So basically out of mere luck. And the output is actually correct! Here is the flow of how textures are used:

Texture is used during shading and namely the function called is sample_textures:

https://github.com/facebookresearch/pytorch3d/blob/fc7a4cacc36ecf2b8feb996bdfb1203dee5d836b/pytorch3d/renderer/mesh/textures.py#L1346-L1373

sample_textures packs the texture by first converting it into a list and then concatenating.

https://github.com/facebookresearch/pytorch3d/blob/fc7a4cacc36ecf2b8feb996bdfb1203dee5d836b/pytorch3d/renderer/mesh/textures.py#L1334-L1338

This leads to the correct packed representation for the textures, even though that's not what you intended.

[gkioxari]

Basically, we don't want the computation to ever go there! It should raise an error when you are setting the textures attributes in meshes. So the fix should check the texture shape when you are trying to set it as an attribute.

Thanks for raising this! We need to fix this to ensure everything is setup right. Also, Covid has us talking about textures on a Friday night :'(

[aluo-x]

I will take a closer look at the texture code tonight and create a pull request tonight/weekend.

[gkioxari]

I don't think textures is relevant here. I envision this being a fix in Meshes when setting the textures attribute. Something along those lines

def __setattr__(self, name, value):
    if name=="textures":
        check whether value and meshes have the same shape
        if shape_is_wrong:
            raise ValueError
        else:
            super().__setattr__(name, value)
    else:
        # We don't want users to change any other attribute
        # apart from textures
        raise ValueError("You cannot set any attribute")

[aluo-x]

That makes a lot of sense. I will tackle this once I complete my current round of 'unreleated' debugging.