Pytorch3d: join_meshes_as_batch() only renders first mesh

[alex-helm]

Hi! join_meshes_as_batch() seems to only render the first mesh in the list of meshes given to it as input by itself on a white background. I checked in other software and the meshes seem fine especially since join_meshes_as_batch() always renders the first mesh correctly no matter which is first. Here is my code:

sys.path.append(os.path.abspath(''))

device = torch.device('cuda:0')
torch.cuda.set_device(device)

mesh1 = load_objs_as_meshes(['./data/mesh1.obj'], device=device)
mesh1_texture = mesh1.textures.maps_padded()

mesh2 = load_objs_as_meshes(['./data/mesh2.obj'], device=device)
mesh2_texture = mesh2.textures.maps_padded()

mesh_batch = join_meshes_as_batch([mesh1, mesh2], include_textures=True)

R, T = look_at_view_transform(dist=500, elev=30, azim=90)
cameras = OpenGLPerspectiveCameras(device=device, R=R, T=T)

raster_settings = RasterizationSettings(
image_size=512,
blur_radius=0.0,
faces_per_pixel=1,
)

lights = PointLights(device=device, location=[[0.0, 0.0, -3.0]])

renderer = MeshRenderer(
rasterizer=MeshRasterizer(
cameras=cameras,
raster_settings=raster_settings
),
shader=TexturedSoftPhongShader(
device=device,
cameras=cameras,
lights=lights
)
)

image = renderer(mesh_batch)
plt.figure(figsize=(10, 10))
plt.imshow(image[0, ..., :3].cpu().numpy())
plt.grid("off");
plt.axis("off");
plt.savefig('./data/scene.jpg')

The same issue occurs when I load verts and faces from the .obj files with load_obj() and create Textures and Meshes objects myself such as in issue #175. Please let me know what I'm missing. Thank you!

[bottler]

join_meshes_as_batch creates a batch of two or more meshes. The line image = renderer(mesh_batch) renders each mesh as a separate image, in image[0, ...] and image[1, ...]. There is no built-in way yet to make a single mesh as the union of two meshes or render a single scene containing multiple meshes.

[gkioxari]

If you need to render two different meshes on the same image then you have to construct a single mesh to be the union of the two meshes. This is super easy to do

If mesh1 = (verts1, faces1) and mesh2 = (verts2, faces2), then construct a mesh = (verts,faces) where

verts = torch.cat([verts1, verts2], dim=0)
faces = torch.cat([faces1, faces2 + verts1.shape[0]], dim=0)

Note that you need to add verts1.shape[0] to faces2 before concatenating so that they index to the correct part of the new mesh.

[bottler]

In this case, it looks like texture maps are in use, so it may also be necessary to merge them into a single image and recompute uv coordinates.

[gkioxari]

Correct, you would have to take care of the textures and any indices there as well.

[nikhilaravi]

Joining two meshes into a single mesh is now supported using the join_mesh function added in https://github.com/facebookresearch/pytorch3d/commit/e053d7c45609ab25e345ac277a89232bfede8e90.

Note currently this only supports joining the verts/faces and not the textures. We plan to add more comprehensive texture support soon.