Azure-kinect-sensor-sdk: The orientation quaternions of CLAVICLE and SHOULDER joints may not right when begin tracking with T-pose

[wqs111000]

Describe the bug

When tracking a body in T-pose, the estimated orientation quaternions of CLAVICLE and SHOULDER joints sometimes may not in conformity with the reality. To facilitate understanding, I convert these quaternions to Euler angles. Sometimes their relative rotation along x-axis may be too large, as to a real body.

To Reproduce

1. One person stands in front of Kinect in T-pose
2. Start body tracker of Kinect and record the information of tracked body.
3. The person adjust his pose.

Expected behavior

The estimated body is consistent with the actual human body even not accurate . Accurate body information is best.

Desktop (please complete the following information):

• OS with Version:Win 10
• SDK Version: body tracking v0.9.5
• Firmware version: RGB camera firmware:1.6.102, Depth camera firmware:1.6.75

[qm13]

@wqs111000 could you please make a recording using the k4arecorder tool that demonstrates the issue. If you can provide me with your email address I can connect with you to get a signed data release form and the recording. Also could you report the quaternion values you see prior to the issue and after the issue.

[wqs111000]

@qm13 .Thanks for your reply, my e-mail is [email protected]. Looking forward to your mail.

[wqs111000]

For example:
K4ABT_JOINT_CLAVICLE_LEFT=4, K4ABT_JOINT_SHOULDER_LEFT=5
Body frame count:1
Joint[4]: Position[mm] ( 11.484035, -574.254089, 2395.017578 ); Orientation ( 0.786721, -0.590111, -0.019147, 0.180205 );Confidence( 2 );
Joint[5]: Position[mm] ( 149.278839, -529.104309, 2368.094238 ); Orientation ( 0.191629, 0.971977, -0.080498, 0.109815 );Confidence( 2 );
Convert quaternions to Euler angles:
Joint[4]:Eular angles [xyz] (-71.811, -14.052, 15.604)
Joint[5]:Eular angles [xyz] (156.595, 10.53, 11.654)
More details can mail to you with a pre-recorded video.

Their relative rotation along x-axis are about 220 degrees(or 160 degrees) which would cause unreal distortion around the shoulder.
Of course I can do some adjustment by myself. But I think this maybe a bug or a false estimate for Kinect tracker.Maybe some constraints can be added to avoid consequences with unattainable states of man.

[Zumra]

Joint[4]: Position[mm] ( 11.484035, -574.254089, 2395.017578 ); Orientation ( 0.786721, -0.590111, -0.019147, 0.180205 );Confidence( 2 );
Joint[5]: Position[mm] ( 149.278839, -529.104309, 2368.094238 ); Orientation ( 0.191629, 0.971977, -0.080498, 0.109815 );Confidence( 2 );

@wqs111000 thank you for providing the example of Joint[4] and Joint[5] values. Based on them we can evaluate the following quantities.

1. pose of the clavicle w.r.t. camera:

p4 = [ 0.9343    0.3061   -0.1826    0.0115
-0.2609    0.2386   -0.9354   -0.5743
-0.2428   -0.9354    0.3028    2.3950
0         0         0    1]

1. pose of the shoulder w.r.t. camera:

p5 = [0.9629   -0.1144    0.2443    0.1493
-0.1986   -0.9136    0.3548   -0.5291
0.1826    0.3548   -0.9024    2.3681
0         0         0    1]

1. pose of the shoulder w.r.t. clavicle:

shoulder_to_clavicle_transform = inv(p4) * p5 =
[1.1706    0.0923   -0.0712    0.1413
-0.5843   -0.1040    0.9449   -0.0396
-0.2633    0.9244   -0.1185   -0.0978
0         0         0    1]

1. given shoulder_to_clavicle_transform, if we extract rotation and convert it to Euler angles we get:

euler_angles = [97.1451    4.0858    4.5069]

This means in your example there is rotation in shoulder which is mostly around x axis.
Please take a look at the joint coordinate axis explained here.
Is it possible hands were not facing downward in the example you've captured and maybe hands were rotated so palms were facing the camera?
Can you post a screen shot of BodyTracking output frame which corresponds to the suspicious orientation values you mention?

Thanks,
Ivan

[wqs111000]

@Ivan .Thanks for your detailed reply. I will check these results and review my codes.

The picture below was captured from the video recorded by k4arecorder tool, which corresponds to the suspicious orientation values.

Some other pictures about the real body and the rigged body are included in the zip file below.
TposeTestCase.zip

[wqs111000]

I used a project based on the jump_analysis_sample to playback the T-pose video with two different versions.The two different version of body tracker gave obviously different orientation quaternions of SHOULDER joints as you can see below.Compared to the picture of the standard T-pose body, it seems the result of the new version is wrong .
body tracker version 0.9.3：

body tracker version 0.9.5：
I'm still working on why the new added six joints are not shown after I updated the project.

https://docs.microsoft.com/en-us/azure/Kinect-dk/body-joints

[wqs111000]

The orientation quaternions of SHOULDER joints are really unstable.
In another test video, the body stayed in standard T-pose at first. As the figure below shows, the result seems wrong.

After some motions , back to standard T-pose. As the figure below shows, the result seems right.

Test with Body tracker version 0.9.5.

[wqs111000]

The rotation in shoulder around x axis w.r.t. clavicle is too big (more than 90 degrees) which results in twist distortion of the rigged body.

[Zumra]

@wqs111000 This is not intended behavior and we are investigating.

[qm13]

We have reproduced the bug and are actively working on a fix.

[Zumra]

@wqs111000 Thank you for reporting the issue and for your help in resolving it. We have addressed it, and the fix will be available in the next release.

[qm13]

Fixed in v1.0.0