Pytorch3d: How can I get 3D vertices consistent with image field？

[TheshowN]

What I'm doing now is to reconstruct a 3dmm face model from a single face image with Pytorch3D, and I want to sample texture from the original image. So I need to get the vertices in the image field, which have the same X and Y coordinates with the corresponding image pixels. But such vertices seem to be encapsulated in render().

Thus, I want to ask if there is an interface to sample textures from the original image, or how can I get vertices with the same X and Y coordinates as the original pixels?

Thanks in advance！

[nikhilaravi]

@TheshowN if I understand correctly, you want to get the value of the 3D vertices projected on to the XY plane?

You can easily do this by applying the transform from the renderer.rasterizer.transform - this will transform the input mesh and project it onto the image plane.

mesh_on_screen = renderer.rasterizer.transform(mesh)

Issue https://github.com/facebookresearch/pytorch3d/issues/353 is about the same question.

[AmosZhu]

Clearly, the MeshRasterizer did all this work, but can the Pytorch3D add an entry to access the mesh_on_screen and the visibility on the corresponding view as well.
To get the mesh_on_screen I can do renderer.rasterizer.transform(mesh)

And for the visibility test, I can do something like

raster_settings = RasterizationSettings(
    image_size=256, 
    blur_radius=0.0,
    faces_per_pixel=1, 
)
rasterizer = MeshRasterizer(
    cameras=cameras, 
    raster_settings=raster_settings
)

fragments = rasterizer(meshes)

But obviously, it calls the transform twice so it's a bit waste, if we can get a return when we do forward rendering that will be much more convenient.
The reason to do this is to add loss for the photometric consistency.

[gkioxari]

We do not expose the transformed vertices, you are right. However, the modular implementation in PyTorch3D allows you to write your own custom renderer and rasterizer class that will allow you to expose whatever you wish for your project based on its needs. This is the beauty of a modular implementation and the intended use in PyTorch3D; namely users should write custom classes to expose and perform whatever they need while we take care of the heavy lifting of the rasterization process, etc.
This way the transform of the vertices will only happen once!