Pytorch3d: SfMPerspectiveCamera requires clarification

[shahabty]

Hello,
I appreciate it if you could explain SfMPerspectiveCamera focal_length and principal_point coordinate system in detail.
I am trying to render a 3D object with the following code:

    verts, faces_idx, _ = load_obj(self.data[x])
    faces = faces_idx.verts_idx
    verts_rgb = torch.ones_like(verts)[None]  # (1, V, 3)
    textures = Textures(verts_rgb=verts_rgb.to(self.device))
    mesh_obj= Meshes(verts=[verts.to(self.device)],faces=[faces.to(self.device)],textures=textures)

    R, T = look_at_view_transform(1000*721, 30, 60, device=device)
    K = torch.tensor([[721,0.,609],[0.,721,172],[0.,0.,1.]]).to(device)
    cameras = SfMPerspectiveCameras(focal_length = ((721,721),),principal_point = ((0.5,0.5),),device = device,R= R,T=T)
    self.cameras = cameras
    blend_params = BlendParams(sigma=1e-4, gamma=1e-4)
    raster_settings = RasterizationSettings(image_size=256, blur_radius= 0.0, faces_per_pixel=100,bin_size = 0)
    self.silhouette_renderer = MeshRenderer(rasterizer=MeshRasterizer(cameras=cameras, raster_settings=raster_settings),shader=SoftSilhouetteShader(blend_params=blend_params))

silhouette = self.silhouette_renderer(meshes_world=mesh_obj)

Then I visualize the silhouette. However, whenever I change the principal_point values to anything other, than zero, the rendered image contains no object. Based on my observation, changing principal_points move the object somewhere out of the boundary of the rendered image. I assume that the coordinate system is different from what I imagine. But I couldn't find any documentation about this.

Thank you,

[gkioxari]

The camera transforms happen in NDC space. These notes might be helpful in understanding the transforms that occur. If your R and T transform the 3D mesh outside the NDC range, then the rendering output will be empty.

We plan on adding support for the SfMPerspectiveCamera to be provided in screen space in the future, and the conversion to NDC space will happen internally. For now though, make sure that all your transforms are NDC space compliant.

[shahabty]

@gkioxari Thank you for your answer. Do you mean that camera intrinsics should remain the same? Could you please provide a code snippet or a guideline to fix this problem in my code?

[gkioxari]

Issue https://github.com/facebookresearch/pytorch3d/issues/171 might be of help.

[shahabty]

@gkioxari Thank you.
However, I still couldn't understand how to use principal points along with focal length.

[gkioxari]

Hi @shahabty

I am not sure I get your question. If (fx, fy) are the focal lengths and (px, py) is the principal point, then a point (X, Y, Z) in the 3D space will be mapped to point (x, y) in image plane as follows (this is the definition of perspective cameras)

x = fx X/Z + px
y = fy Y/Z + py

The cameras currently assume that the focal lengths and the principal points are provided in NDC space. We will add support shortly to convert screen space to NDC space. In essence, our fix will convert the screen space focal lengths and principal points, provided by the user, to NDC space.

For your case, I assume your principal point and focal length are in screen space. The conversion to NDC space is easy. If your image has size (H, W), and if half_W = W/2, half_H = H/2 then

px_ndc = (px - half_W) / half_w
py_ndc = (py - half_H) / half_H
fx_ndc = fx / half_W
fy_ndc = fy / half_H

So for now, provide the _ndc parameters to the SfM camera and things should work, if your initial parameters in screen space are the desired ones.

[gkioxari]

We have fixed this now with https://github.com/facebookresearch/pytorch3d/commit/57a22e7306db24140cb2133aa12a613cbf971c4c

Read more here: https://github.com/facebookresearch/pytorch3d/blob/master/docs/notes/cameras.md

To recap, PyTorch3D cameras assume parameters are provided in NDC space. We have updated cameras such that SfMPerspective and SfMOrthographic can also accept parameters in screen space. Read the note and the code for more details.