Openpose: 3d skeleton from keypoints

[toinsson]

I would like to get a 3d model out of the keypoints extracted, but the current format only provides pixel coordinates in the 2d image associated with confidence scores. From the way the algorithm works, I suspect it is not straightforward but I might be wrong. Is there is a way to get such information from openpose. If not, is it possible to get some pointers towards achieving such goal?

[jijup]

I work on a similar problem, but on RGBD. In case, if you have depth information, you can use the camera intrinsics and extrinsics to map the 2D pixel coordinates to corresponding 3D points in the camera space. I use realsense SDK, which provides the 2D to 3D projection functions (https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense2/rsutil.h). Once you have the 3D points, you may use the joint names (provided by openpose) to construct the skeleton.

[svarnypetr]

We are trying to solve a similar problem. The general idea @jijup described is what we want to do.

However we are struggling at aligning the frames. I am able to let OpenPose read a RGB stream and do its pose detection, and let the PyRealsense read the depth frames. But we did not find a way how to make it certain that the frame where OpenPose detected a pose is the same as the one we have depth the depth data (the same w.r.t. time, obviously I need to use a transform in order to get from RGB to Depth) .

I tried installing PyOpenPose for this but failed with Bus error or tried in various ways to send single frames to OpenPose for pose estimation trough a python pipeline or mimicking an IP camera. The only working solution I managed to get was saving frames and reading them from the disc which was obviously very slow.

[gineshidalgo99]

1 option is the 3-d reconstruction demo in doc/3d_reconstruction_demo.md The second option is Kinect or other depth sensors.

@svarnypetr For that purpose or any custom input, you should create your own code by using the examples/tutorial_wrapper/* examples

Feel free to ask again if you need further clarification.

[tolgasaglik]

@jijup Hi, I use SR300 camera with librealsense2 and want to get body node z values as you did.
Could you explain a little bit further how you achieved extracting this information?

[svarnypetr]

@tolgasaglik is this still relevant or did you already manage to solve it? I noticed your question just now but here is the code we used with a newer Realsense camera, but should work similarly with SR300: https://github.com/svarnypetr/pps

[highway007]

@svarnypetr I did some similar work refer yours,but the xyz points was shaked(I put them into unity and with kinect v2).I wonder how it performed in your project ? Can you show me some demo gifs or videos? Thx ; )

[svarnypetr]

@highway007 I think such a discussion is out of the scope of this issue and I also did not use kinect v2 nor try to put it to Unity, there might be many differences there. I do not have videos yet to share but they would not be helpful as we did not capture on video the 3D position of the keypoints.

[highway007]

@svarnypetr Yeap,I think you are right.Thx anyway.

[DroneMeOut]

http://files.is.tue.mpg.de/black/papers/BogoECCV2016.pdf