Openmvg: Rotation registration with android phone euler angles

[EthanGreen75]

Hi, Dr. Pmoulon,
As what I mentioned before in the issue #432 and issue #388, I want to use the yaw, pitch, roll angles(or rotation matrix) and GPS information of my android phone together to solve the registration problem. The first thing is about the rotation registration.
I've been worked with this problem with several weeks, but the result didn’t work out. So I want your help with my work sincerely. Before I illustrate my method, I have some questions to ask about the whole transformation.

Questions
1. In OpenMVG, what is the multiplication order of the euler angles? Yaw, pitch, roll have too may definitions in different fields, so I want to call them AngleZ, AngleY, AngleX instead, which stands for the rotation around axis Z, axis Y, axis X respectively.
In consideration of the code of the function RefineRTS() in the project of registration with GPS, I presume that the order is Rx* Ry* Rz. So, after getting the rotation matrix of each camera after SFM procedure, first I get the transpose of the R because I think the R^Tstands for the rotation matrix about the three coordinates. And then I decompose R^T to get the three euler angles in the inverse way how the rotation formed by the Rx* Ry* Rz order.
The reason why I want to know about the order is that I think in SFM space and true world space, If we want to get the right transformation matrix between a pair of cameras, the two cameras(one in SFM space and the other in true world space) must have the same order of euler angle matrix multiplication. If I can get the euler angles of X, Y, Z axis of the camera in the true world, I can get the right transformation matrix between these two cameras.
2. The transformation matrix between android phone's frame and camera's frame. As the picture shows below, the phone’s axis and camera’s axis are different.

I can get the inR which stands for the rotation between android phone’s frame and world frame. I notice that the x axis and y axis are swapped and I use the android interface remapCoordinateSystem(inR, AXIS_Y, AXIS_X, outR) to get the rotation matrix between camera frame and world frame.
Then I decompose the outR in the inverse way how the rotation formed by Rz* Ry* Rx order. Actually, the initial order of the inR is Rz* Rx* Ry, but when I try to use this order to decompose I get the wrong angle. I presume that It’s because I just swap the x axis and y axis. So we get the three angles which stands for the camera frame’s rotation about the world frame.
3．What is the right transformation between two rotation matrix?
If we get the two rotation matrix in SFM space and true world space, we should use left multiplication or right multiplication to get the transformation matrix? The left multiplication is R_real= R_trans* R_SFM and the right multiplication is R_real= R_SFM* R_trans. Again, in consideration of the GPS registration project, I think it’ s right multiplication.

Procedures
Now, I would explain my method below as precise as I can.
1. Get all the rotation matrix of cameras from the SFM output and get their inverse.
2. Get the three euler angles of the phone in the true world space and transform them to the camera’s three euler angles. And then multiply in the order Rx* Ry* Rz to get the rotation matrix of each camera.
3. Use the formulation R_real= R_SFM* R_trans to get the R_trans which is the transformation matrix from SFM space to real world space.
Actually, for each pair of these two cameras, we can get an R_trans. So, I think there are two rules to verify whether the R_trans is right.
1) All the R_trans should have almost same elements of the matrix. Or if we decompose the R_trans into three euler angles, these three angles should have almost the same value.
2) If we take one of the R_trans to multiply all the other camera’s rotation matrix in SFM space（R_SFM）we can get the corresponding rotation matrix in real world space(R_real).
If R_trans satisfies these two rules, we can think that it is right.
4. Use R_trans to transform the cloud points. I think that the R_trans is the transformation between the cameras, we can’t use R_trans to transform the points directly. In consideration of the method used in GPS registration, maybe we should use R_trans^T to multiply the points. I don’t know it is right or not.

Experiments Results & Conclusion
After I take all the procedures above, the results turn to be wrong. The two right rules for R_trans couldn’t be satisfied. I learned everything from scratch by myself and I know that there must be something I understand wrongly or miss. All the knowledge are really fragmented and complicated. I need your help and hope you can figure me out what is the right thing to do.

[whuaegeanse]

Actually, this is about the transformation of the coordinates system.
(1) the android rotation matrix
Mat3 Rph = RotationAroundX(D2R(dThetaX)) * RotationAroundY(D2R(dThetaY)) * RotationAroundZ(D2R(dThetaZ));
where the Rph is the rotation matrix in your android phone, and the dThetaX ,dThetaY and dThetaZ are the three degree with rotation axis X , Y and Z respectively, in the order of X Y Z.
(2) the computer vision rotation matrix
openmvg use ceres::AngleAxisToRotationMatrixto compute rotation matrix. This rotation system is different from your android phone rotation system.
(3) the transformation

Mat3 Rcv = (Rph*RotationAroundX(D2R(180))).transpose();
Mat3 Rph = Rcv.transpose()*(RotationAroundX(D2R(180)).transpose());

[EthanGreen75]

@whuaegeanse Yes, it is. I've also read about your issue. It seems that you also did something about the rotation registration. Do you have any suggestions about my problem? Or do you think there is a better way to do the rotation registration?

[EthanGreen75]

@whuaegeanse
Thank you for replying me so quickly. I am so surprised and glad that you are also a Chinese. You are in Chengdu and I am in Xi'an. However, I still have some confusion about the transformation.

1. In my point of view, Rcv is the camera rotation matrix in the true world, is that right? First we get phone's rotation matrix Rph in the X,Y,Z order, and then multiply Rx(180), finally the Rcv is the inverse of it. I don't know how the transformation work. Could you explain about the principle more detailedly?
   And I also tried the Rph = R_SFM * R_trans to get the R_trans for each pair of cameras. They vary a lot. I don't think the R_trans is the right one.
2. As you mentioned, Rph is in the order of X, Y, Z. Do you think this order is the truth for OpenMVG?
   3.How to compute the R_trans, which is the transformation between the cameras in SFM space and true world space.
   I am just so confused about the whole transformation procedure. I've tried many ways but failed.

[whuaegeanse]

1. Did you have a trial?
2. In openMVG, the rotation axis-angle is used and in your android phone, Euler Angle is used. They are different. http://www.geometrictools.com/Documentation/EulerAngles.pdf,https://en.wikipedia.org/wiki/Rotation_matrix#Conversion_from_and_to_axis-angle

[whuaegeanse]

When you use different software or opensource project, please be careful the rotation system. It's better using rotation matrix for the conversion.

[EthanGreen75]

@whuaegeanse Thank you for the two links for the euler angles and angle axis.
Actually I did use rotation matrix to calculate the transformation. As shown in my 4 procedures, the reason why I want to know the order is that I think in SFM space and real world space, the rotation matrix should have same multiplication order. I want to construct the rotation matrix in real world as the order in the OpenMVG.
OpenMVG use the ceres to perform BA, so it transform the rotation matrix of each pose to angle axis to execute BA. I want to directly use the rotation matrix of each pose to calculate the transformation. Need I compute the angle axis of the rotation in the true world and compute relative relationship between two angle axis?

[shuguohui]

@whuaegeanse in GPS/INS system,the angle order is yaw(Z) pitch(Y) roll(X)?
I finish Phi Omega kappa,as we have the image dataset. But I don't validate the GPS/INS,because all the images in my hand are UAV's INS,not the camera's INS.

[whuaegeanse]

Please contact me for free. My qq acount is 514198609.

[EthanGreen75]

@pmoulon
After talking with whuaegeanse, I've corrected something wrong.
1.The right way to construct camera rotation matrix in the world.
I used to think that I have to construct the rotation matrix in the same order of OpenMVG, But it turns out that every euler angle is unique, especially for constructing intrinsic rotation matrix. If we chagne the order at random, we can't get the right gesture of the phone or camera.
So I still use the android api of getRotationMatrix, it returns the rotation matrix around the NEG(North, East, Gravity) frame. The relationship between the camera frame and phone frame can be transformed as follows:
Rcv = Rph * Rx(180) * Rz(-90), Rcv means the camera's rotation matrix around NEG frame, Rph means the phone's rotation matrix around NEG frame. And I tested the Rcv in the project SFM_From_Known_Poses and it turns out to be right.

2.Rotation registration still remains unsettled.
As I've said before, Two rules should satisfied for two sets of cameras' rotations. But the experiments show that Even if I've get camera's right rotation matrix in SFM space and world space, the transformation matrix between each pair of rotation matrix varies a lot.
Maybe this kind of transformation between two rotation matrix is wrong, or I can use another kind of way of transformation, like quaternion or angle aixs?
Hope you can give me some hint for that. I've been struggled with this for too long.

[EthanGreen75]

@pmoulon
I've known the way about rotation registration. After we get the two rotation matrix in SFM space and true world space. We should use left multiplication to get the transformation rotation matrix. That is , R_Trans * R_SFM = R_Real. And in this situation, the two rules could be satisfied. Because that's extrinsic rotation, the transformation matrix should be in the left. Some dataset proved this. I need to go on to finish my whole registration procedures.

[pmoulon]

Happy to hear that you found the right way to do it, it's always better to use full rotation matrices.
I got some days off and was unable to add answer in this thread.
BTW I was happy to see that the community (@whuaegeanse @shuguohui) helped @EthanGreen75.