Pytorch3d: Convert camera extrinsic openCV to R and T matrix of SfMPerspectiveCameras

[hiro050696]

camera_pose  = np.array([[ 0.10624042,  0.99397693,  0.0268858,   0.2087519 ],
                            [-0.68876547,  0.05406284,  0.72296565,  0.61393758],
                            [ 0.71715765, -0.09532618,  0.69036065,  3.16445006],
                            [0.0        ,           0.0,        0.0,        1.0]])


def getNewCameraPose(camera_pose=camera_pose):
    rot_X       = np.array([[ 1.0, 0.0, 0.0, 0],
                            [ 0.0, -1.0, 0.0, 0],
                            [ 0.0, 0.0, -1.0, 0],
                            [0, 0, 0, 1]])
    rot_X_1     = np.array([[ 1.0, 0.0, 0.0, 0],
                            [ 0.0, 0.0, 1.0, 0],
                            [ 0.0, -1.0, 0.0, 0],
                            [0, 0, 0, 1]])
    A = np.dot(rot_X,camera_pose)
    A = np.dot(A,rot_X_1)
    camera_pose = np.linalg.inv(A)
    return camera_pose

camera_pose = getNewCameraPose(camera_pose=camera_pose)
R             = torch.Tensor([camera_pose[:3,:3]])
T             = torch.Tensor([camera_pose[:3,3]])

After using OpenCV cabliration, i got a 4x4 extrinsic matrix. I want to convert it to 2 matrix R and T form OpenCV coordinate to Pytorch3D coordinate, those matrix are used in SfMPerspectiveCameras projection function.
I tried a lot of approach but it didnt work correctly, maybe the problem is related to right-hand and left-hand coordinate system.

How can I do it?

[nikhilaravi]

So to clarify you used the R & T (from the code example above) with SfMPerspectiveCameras and did got get the correct rendering/transformation result?

There are two things you should check:

• Internally in the Cameras class we use the Transforms3D class for the transforms. This assumes that transformations are applied on inputs which are row vectors. Does this apply for your case with the R matrix? https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/transforms/transform3d.py#L114-L133
• The coordinate system conventions are aligned. Here is a reference for the convention used in PyTorch3d:

[gkioxari]

Here is another illustration of the coordinate systems if it helps

[gkioxari]

We have provided a more comprehensive description of our cameras along with coordinate systems. You can read more in our notes here:

https://github.com/facebookresearch/pytorch3d/blob/master/docs/notes/cameras.md

and also in the codebase. Let us know if you need any more info.

[nikhilaravi]

@hiro050696 we will close this issue. If you have further questions feel free to reopen it.

[BostonLobster]

@hiro050696 Have you solved this problem? I'm also trying to convert extrinsics from ScanNet (which is in OpenCV coordinate system), but feeding converted [R, T] into SfMPerspective got empty rendering results.

[jbohnslav]

I'm also working on this problem. Has anyone figured it out?

Looking at the opencv documentation, I think the reason for the issue is the coordinate systems. OpenCV uses a X-right, Y-down, Z-out (positive) coordinate system for pinhole camera model

As Nikhila and Georgia showed in the really nice docs, Pytorch3D uses a X-left, Y-up, Z-out coordinate system.

[pengsongyou]

Hi,

For those who are still looking for the solution, here is my solution: https://github.com/facebookresearch/pytorch3d/issues/522#issuecomment-762793832.
Hope it helps.

Best,
Songyou